Sep 16 12:46:16.681462 ip-10-0-135-92 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 12:46:16.681474 ip-10-0-135-92 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 12:46:16.681481 ip-10-0-135-92 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 12:46:16.681696 ip-10-0-135-92 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 12:46:26.035047 ip-10-0-135-92 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 16 12:46:26.035059 ip-10-0-135-92 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8c1e1f0e7bfd4c3ea7a5a68ba16f5c05 --
Sep 16 12:47:58.702928 ip-10-0-135-92 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:47:59.126027 ip-10-0-135-92 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:59.126027 ip-10-0-135-92 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:47:59.126027 ip-10-0-135-92 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:59.126027 ip-10-0-135-92 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:47:59.126027 ip-10-0-135-92 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:47:59.126681 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.126090    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:47:59.129925 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129909    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:59.129925 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129924    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129928    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129931    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129933    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129936    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129939    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129941    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129944    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129946    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129949    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129952    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129954    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129957    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129960    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129962    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129965    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129967    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129969    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129972    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129974    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:59.129994 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129978    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129980    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129983    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129986    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129988    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129991    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129994    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129997    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.129999    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130002    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130005    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130008    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130011    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130013    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130015    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130018    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130020    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130023    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130025    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130027    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:59.130468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130030    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130032    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130035    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130037    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130040    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130044    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130048    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130051    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130054    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130057    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130060    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130063    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130066    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130068    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130071    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130074    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130077    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130080    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130084    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:59.131229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130087    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130090    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130093    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130095    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130097    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130100    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130102    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130104    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130107    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130110    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130112    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130114    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130118    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130122    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130124    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130127    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130131    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130135    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130138    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:59.131795 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130141    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130143    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130147    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130150    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130152    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130155    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130158    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130160    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130164    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130167    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130170    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130172    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130175    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130178    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130181    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130184    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130186    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130189    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130192    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:59.132256 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.130195    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:59.133844 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133828    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:59.133844 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133844    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133848    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133851    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133854    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133857    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133860    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133863    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133865    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133869    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133871    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133874    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133876    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133879    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133881    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133884    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133886    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133889    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133892    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133894    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133897    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:59.133919 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133899    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133902    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133904    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133907    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133910    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133912    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133915    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133917    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133920    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133923    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133926    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133929    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133932    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133934    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133937    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133939    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133941    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133944    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133947    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133949    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:59.134416 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133951    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133954    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133956    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133959    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133961    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133963    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133966    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133968    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133970    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133974    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133976    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133979    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133981    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133984    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133986    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133990    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133992    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133995    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.133998    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:59.134946 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134000    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134002    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134006    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134009    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134011    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134014    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134018    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134021    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134024    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134027    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134030    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134032    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134034    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134037    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134040    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134042    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134046    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134049    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134053    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:59.135406 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134056    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134059    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134061    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134064    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134067    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134070    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134074    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134077    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134080    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134082    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134085    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134087    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134090    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134093    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134096    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134098    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134101    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134103    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134106    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134109    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:59.135888 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134482    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134487    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134489    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134492    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134495    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134498    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134501    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134503    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134506    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134508    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134511    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134513    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134516    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134518    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134521    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134523    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134526    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134529    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134532    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134534    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:59.136386 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134538    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134541    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134543    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134546    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134548    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134551    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134553    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134556    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134558    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134561    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134563    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134566    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134569    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134571    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134574    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134576    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134579    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134581    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134583    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:59.136885 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134586    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134588    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134590    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134593    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134595    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134598    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134600    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134602    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134605    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134607    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134610    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134612    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134615    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134619    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134622    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134624    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134626    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134629    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134631    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134634    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:59.137334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134636    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134638    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134660    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134664    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134667    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134670    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134672    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134675    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134682    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134686    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134690    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134694    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134696    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134699    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134702    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134704    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134706    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134709    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134711    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134714    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:59.137828 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134716    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134719    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134721    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134724    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134727    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134730    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134733    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134735    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134738    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134740    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134744    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134747    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134751    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134754    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134757    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134760    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134763    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134765    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134767    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:59.138293 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.134770    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134836    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134871    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134878    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134882    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134885    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134888    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134893    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134896    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134899    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134902    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134906    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134909    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134912    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134915    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134918    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134920    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134924    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134928    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134931    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134934    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134937    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134940    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134943    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:47:59.138763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134947    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134950    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134953    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134956    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134959    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134963    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134965    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134969    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134972    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134975    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134978    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134981    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134984    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134987    2575 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134989    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134995    2575 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.134998    2575 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135001    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135004    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135007    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135010    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135013    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135016    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135019    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135022    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:47:59.139333 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135025    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135028    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135031    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135034    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135037    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135040    2575 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135043    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135046    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135049    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135052    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135055    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135057    2575 flags.go:64] FLAG: --help="false"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135060    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135063    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135066    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135068    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135071    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135074    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135077    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135080    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135083    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135085    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135088    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135091    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:47:59.140031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135094    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135097    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135100    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135103    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135105    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135108    2575 flags.go:64] FLAG: --lock-file=""
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135111    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135114    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135117    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135122    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135125    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135128    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135131    2575 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135133    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135137    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135139    2575 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135142    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135146    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135149    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135153    2575 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135155    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135158    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135161    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135164    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135167    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:47:59.140609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135170    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135172    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135179    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135182    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135185    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135188    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135190    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135196    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135201    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135204    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135208    2575 flags.go:64] FLAG: --port="10250"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135211    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135214    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04a6dd2637f08737e"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135217    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135219    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135222    2575 flags.go:64] FLAG: --register-node="true"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135225    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135228    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135232    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135235    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135237    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135241    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135244    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135247    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135250    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:47:59.141234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135252    2575 flags.go:64] FLAG: --runonce="false"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135255    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135258    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135261    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135264    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135267    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135269    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135272    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135275    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135277    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135281    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135283    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135286    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135289    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135292    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135296    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135300    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135305    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135307    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135311    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135314    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135317    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135319    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135322    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135325    2575 flags.go:64] FLAG: --v="2"
Sep 16 12:47:59.141855 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135328    2575 flags.go:64] FLAG: --version="false"
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135334    2575 flags.go:64] FLAG: --vmodule=""
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135337    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.135340    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135895    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135900    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135904    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135915    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135920    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135923    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135926    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135929    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135932    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135935    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135937    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135940    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135943    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135945    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135948    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135950    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:59.142465 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135953    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135955    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135959    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135962    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135965    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135968    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135970    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135973    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135975    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135977    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135980    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135982    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135984    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135987    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135990    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135993    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135995    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.135997    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136000    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136002    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:59.142988 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136005    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136007    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136010    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136012    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136014    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136017    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136019    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136021    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136024    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136026    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136029    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136031    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136034    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136036    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136040    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136042    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136045    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136048    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136050    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136053    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:59.143488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136055    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136058    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136060    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136063    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136066    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136070    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136073    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136076    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136079    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136081    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136084    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136086    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136089    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136091    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136093    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136096    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136099    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136101    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136103    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:59.144011 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136106    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136108    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136111    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136113    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136115    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136118    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136120    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136124    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136126    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136129    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136131    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136133    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136136    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136138    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136141    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136143    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136145    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136148    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136151    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136154    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:59.144471 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136157    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136159    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136162    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.136164    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.136957    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.144328    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.144342    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.144355    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.144361    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144414    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144418    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144422    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144424    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144427    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144430    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:59.144972 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144433    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144435    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144438    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144441    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144444    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144446    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144449    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144452    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144454    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144458    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144462    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144465    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144468    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144471    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144474    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144477    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144479    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144482    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144485    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144488    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:59.145349 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144490    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144493    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144495    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144500    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144502    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144505    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144508    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144511    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144513    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144516    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144518    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144521    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144523    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144526    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144528    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144531    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144533    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144536    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144538    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:59.145900 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144542    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144546    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144548    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144551    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144553    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144555    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144558    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144561    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144563    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144566    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144568    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144571    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144573    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144576    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144578    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144580    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144583    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144587    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144590    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144592    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:59.146446 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144594    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144597    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144600    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144602    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144605    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144607    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144609    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144612    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144615    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144617    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144620    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144622    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144624    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144627    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144630    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144632    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144634    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144638    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144665    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144669    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:59.146944 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144672    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144674    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144677    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144680    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144683    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144685    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144688    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144690    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144693    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144696    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144699    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144702    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144705    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144707    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.144712    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:59.147433 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144817    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144822    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144824    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144827    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144830    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144833    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144835    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144838    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144840    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144843    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144845    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144848    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144850    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144853    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144855    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144858    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144860    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144863    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144865    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144868    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:47:59.147850 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144870    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144872    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144875    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144877    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144881    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144885    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144888    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144891    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144894    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144897    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144899    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144901    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144904    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144906    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144909    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144911    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144914    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144917    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144920    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144922    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:47:59.148351 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144924    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144927    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144929    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144931    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144934    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144936    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144939    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144941    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144944    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144946    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144949    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144951    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144955    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144958    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144961    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144963    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144967    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144971    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144973    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:47:59.148992 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144976    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144979    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144982    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144985    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144987    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144990    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144992    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144995    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.144997    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145000    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145002    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145005    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145007    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145009    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145012    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145014    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145016    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145019    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145021    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145024    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:47:59.149445 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145026    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145029    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145031    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145034    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145036    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145038    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145041    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145043    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145046    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145048    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145051    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145053    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145055    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145058    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145060    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145063    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145066    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145068    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145070    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:47:59.150229 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:47:59.145073    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:47:59.150884 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.145077    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:47:59.150884 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.145169    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:47:59.150884 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.147406    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:47:59.150884 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.148456    2575 server.go:1019] "Starting client certificate rotation"
Sep 16 12:47:59.150884 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.148568    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:47:59.150884 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.148619    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:47:59.151446 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.151432    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:47:59.152516 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.152487    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:47:59.167925 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.167898    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:47:59.172034 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.172011    2575 log.go:25] "Validated CRI v1 image API"
Sep 16 12:47:59.172906 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.172892    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:47:59.175034 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.175016    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:47:59.179965 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.179943    2575 fs.go:135] Filesystem UUIDs: map[7996096d-99f2-448b-a79a-b8c232cdcce1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9eeb7983-4afa-4043-b476-0ca9be48e9a8:/dev/nvme0n1p3]
Sep 16 12:47:59.180040 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.179966    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:47:59.186311 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.186203    2575 manager.go:216] Machine: {Timestamp:2026-09-16 12:47:59.183807469 +0000 UTC m=+0.364950113 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097626 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec206ad653fba083bdbba14e3d638dea SystemUUID:ec206ad6-53fb-a083-bdbb-a14e3d638dea BootID:8c1e1f0e-7bfd-4c3e-a7a5-a68ba16f5c05 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:df:31:7c:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:df:31:7c:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:80:32:07:da:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:47:59.186311 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.186307    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:47:59.186418 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.186407    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:47:59.188437 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.188414    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:47:59.188561 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.188440    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:47:59.189951 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.189941    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:47:59.189983 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.189953    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:47:59.189983 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.189960    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:47:59.190735 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.190726    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:47:59.191719 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.191710    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:47:59.194843 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.194832    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:47:59.194944 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.194935    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:47:59.194981 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.194966    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:47:59.194981 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.194976    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:47:59.195042 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.194991    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:47:59.195042 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.195000    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:47:59.195042 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.195019    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:47:59.195340 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.195331    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:47:59.195374 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.195348    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:47:59.197638 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.197624    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:47:59.200719 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.200706    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:47:59.200782 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.200736    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:47:59.202661 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202632    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:47:59.202728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202665    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:47:59.202728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202673    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:47:59.202728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202678    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:47:59.202728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202686    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:47:59.202728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202693    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:47:59.202728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202706    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:47:59.202728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202711    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:47:59.202728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202716    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:47:59.202728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202722    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:47:59.203099 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202738    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:47:59.203099 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.202744    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:47:59.203414 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.203396    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k6v98"
Sep 16 12:47:59.203627 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.203617    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:47:59.203627 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.203627    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:47:59.207396 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.207384    2575 server.go:1268] "Started kubelet"
Sep 16 12:47:59.207486 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.207452    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:47:59.207591 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.207470    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:47:59.207591 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.207529    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:47:59.208155 ip-10-0-135-92 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:47:59.212529 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.212504    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k6v98"
Sep 16 12:47:59.212529 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.212523    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:47:59.214262 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.214243    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:47:59.215110 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.215089    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:47:59.215234 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.215127    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:47:59.215327 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.215304    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:47:59.215465 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.215440    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:47:59.220149 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.220134    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:47:59.220207 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.220153    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:47:59.220323 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.220307    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:47:59.220879 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.220855    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:47:59.220879 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.220860    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:47:59.221013 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.220889    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:47:59.221013 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.221006    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:47:59.221084 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.221017    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:47:59.222865 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.222711    2575 factory.go:55] Registering systemd factory
Sep 16 12:47:59.222865 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.222733    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:47:59.223522 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.223455    2575 factory.go:153] Registering CRI-O factory
Sep 16 12:47:59.223522 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.223472    2575 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:47:59.223522 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.223519    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:47:59.223522 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.223542    2575 factory.go:103] Registering Raw factory
Sep 16 12:47:59.223522 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.223556    2575 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:47:59.224018 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.223994    2575 manager.go:349] Starting recovery of all containers
Sep 16 12:47:59.228389 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.228364    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:59.232926 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.232904    2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-92.ec2.internal\" not found" node="ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.235046 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.235026    2575 manager.go:354] Recovery completed
Sep 16 12:47:59.238907 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.238895    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:59.241509 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.241492    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:59.241560 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.241522    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:59.241560 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.241532    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:59.242077 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.242063    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:47:59.242132 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.242077    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:47:59.242132 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.242095    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:47:59.242132 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.242101    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:59.242849 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.242835    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:59.242896 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.242860    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:59.242896 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.242869    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:59.243468 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.243456    2575 policy_none.go:49] "None policy: Start"
Sep 16 12:47:59.243497 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.243472    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:47:59.243497 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.243482    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:47:59.244438 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.244428    2575 policy_none.go:47] "Start"
Sep 16 12:47:59.278220 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.278205    2575 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:47:59.291368 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.278255    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:47:59.291368 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.278269    2575 server.go:87] "Starting device plugin registration server"
Sep 16 12:47:59.291368 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.278487    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:47:59.291368 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.278500    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:47:59.291368 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.278595    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:47:59.291368 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.278694    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:47:59.291368 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.278702    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:47:59.291368 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.279153    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:47:59.291368 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.279191    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:47:59.320984 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.320962    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:47:59.322285 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.322264    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:47:59.322362 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.322308    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:47:59.322362 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.322327    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:47:59.322437 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.322360    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:47:59.324948 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.324926    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:47:59.385411 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.379313    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:59.385411 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.381613    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:59.385411 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.381641    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:59.385411 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.381668    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:59.385411 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.381691    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.390847 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.390834    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.390900 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.390853    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-92.ec2.internal\": node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:47:59.409181 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.409162    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:47:59.423392 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.423373    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal"]
Sep 16 12:47:59.423453 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.423441    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:59.424196 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.424180    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:59.424261 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.424210    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:59.424261 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.424219    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:59.425348 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.425337    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:59.425488 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.425476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.425521 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.425503    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:59.426048 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.426031    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:59.426110 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.426036    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:59.426110 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.426063    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:59.426110 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.426074    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:59.426110 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.426075    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:59.426110 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.426086    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:59.427281 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.427267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.427318 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.427312    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:47:59.428142 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.428128    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:47:59.428211 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.428149    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:47:59.428211 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.428163    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:47:59.452173 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.452154    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-92.ec2.internal\" not found" node="ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.456533 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.456517    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-92.ec2.internal\" not found" node="ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.509864 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.509834    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:47:59.522120 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.522100    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b59d531774340d51b7b4e079c1cee170-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal\" (UID: \"b59d531774340d51b7b4e079c1cee170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.522196 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.522131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b59d531774340d51b7b4e079c1cee170-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal\" (UID: \"b59d531774340d51b7b4e079c1cee170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.522196 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.522186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/377f3cc72223911dbae51e9c32e9bb13-config\") pod \"kube-apiserver-proxy-ip-10-0-135-92.ec2.internal\" (UID: \"377f3cc72223911dbae51e9c32e9bb13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.610625 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.610603    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:47:59.622658 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.622624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b59d531774340d51b7b4e079c1cee170-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal\" (UID: \"b59d531774340d51b7b4e079c1cee170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.622742 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.622668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b59d531774340d51b7b4e079c1cee170-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal\" (UID: \"b59d531774340d51b7b4e079c1cee170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.622742 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.622693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/377f3cc72223911dbae51e9c32e9bb13-config\") pod \"kube-apiserver-proxy-ip-10-0-135-92.ec2.internal\" (UID: \"377f3cc72223911dbae51e9c32e9bb13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.622742 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.622728    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b59d531774340d51b7b4e079c1cee170-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal\" (UID: \"b59d531774340d51b7b4e079c1cee170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.622742 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.622737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b59d531774340d51b7b4e079c1cee170-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal\" (UID: \"b59d531774340d51b7b4e079c1cee170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.622881 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.622738    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/377f3cc72223911dbae51e9c32e9bb13-config\") pod \"kube-apiserver-proxy-ip-10-0-135-92.ec2.internal\" (UID: \"377f3cc72223911dbae51e9c32e9bb13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.711084 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.711031    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:47:59.755480 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.755461    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.759524 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:47:59.759215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal"
Sep 16 12:47:59.811715 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.811693    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:47:59.912223 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:47:59.912203    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:48:00.012734 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:00.012674    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:48:00.113231 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:00.113207    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:48:00.148677 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.148636    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:48:00.149025 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.148780    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:48:00.149025 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.148797    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:48:00.214199 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:00.214051    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:48:00.217772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.217749    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 12:42:59 +0000 UTC" deadline="2028-06-17 02:20:09.089769742 +0000 UTC"
Sep 16 12:48:00.217840 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.217772    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15349h32m8.872000179s"
Sep 16 12:48:00.315032 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:00.314966    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:48:00.355125 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:00.355093    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59d531774340d51b7b4e079c1cee170.slice/crio-9c3049778e7d854de6ab9ce676534ec80c4f5da9c80ae52f95c2ce5c803743b4 WatchSource:0}: Error finding container 9c3049778e7d854de6ab9ce676534ec80c4f5da9c80ae52f95c2ce5c803743b4: Status 404 returned error can't find the container with id 9c3049778e7d854de6ab9ce676534ec80c4f5da9c80ae52f95c2ce5c803743b4
Sep 16 12:48:00.355559 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:00.355539    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377f3cc72223911dbae51e9c32e9bb13.slice/crio-05806f19e41e43c66f579cede7f726b6c95ae8325072af5ec4ba9260a8a6fb2f WatchSource:0}: Error finding container 05806f19e41e43c66f579cede7f726b6c95ae8325072af5ec4ba9260a8a6fb2f: Status 404 returned error can't find the container with id 05806f19e41e43c66f579cede7f726b6c95ae8325072af5ec4ba9260a8a6fb2f
Sep 16 12:48:00.360763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.360746    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:48:00.415823 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:00.415800    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:48:00.430120 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.430097    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:00.516833 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:00.516804    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:48:00.617373 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:00.617287    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:48:00.718115 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:00.718071    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-92.ec2.internal\" not found"
Sep 16 12:48:00.808022 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.807994    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:00.820541 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.820520    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal"
Sep 16 12:48:00.830726 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.830702    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:48:00.832251 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.832092    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal"
Sep 16 12:48:00.842863 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:00.842848    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:48:01.196155 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.196124    2575 apiserver.go:52] "Watching apiserver"
Sep 16 12:48:01.206457 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.206425    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:48:01.206899 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.206871    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-dw7rd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal","openshift-multus/network-metrics-daemon-6xf6c","openshift-network-diagnostics/network-check-target-l2cs6","kube-system/global-pull-secret-syncer-ncqr5","kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c","openshift-cluster-node-tuning-operator/tuned-2tm9p","openshift-multus/multus-additional-cni-plugins-vhhd8","openshift-multus/multus-pwrvh","openshift-network-operator/iptables-alerter-lkwg9","openshift-ovn-kubernetes/ovnkube-node-bc9vz","kube-system/konnectivity-agent-r5dpw","openshift-dns/node-resolver-4p7dg"]
Sep 16 12:48:01.208621 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.208602    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.209942 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.209918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:01.211553 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.211526    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:01.211711 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.211680    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:01.211852 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.211833    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jjt8l\""
Sep 16 12:48:01.211922 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.211848    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:48:01.211995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.211977    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:48:01.212074 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.211985    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:48:01.212866 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.212844    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:01.212963 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.212931    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:01.213032 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.212962    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:48:01.213104 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.213088    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:48:01.213203 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.213188    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qtnw4\""
Sep 16 12:48:01.214204 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.214185    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.214310 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.214260    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:01.215441 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.215419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.217015 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.216991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.218383 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.218363    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:48:01.218493 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.218365    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:48:01.218493 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.218365    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9rr7p\""
Sep 16 12:48:01.218593 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.218503    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.219875 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.219753    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:48:01.219875 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.219853    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:48:01.220031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.219878    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:48:01.220031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.219928    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2lxbf\""
Sep 16 12:48:01.220031 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.219954    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:48:01.220302 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.220287    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.220416 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.220398    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:48:01.221351 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.221330    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:48:01.223276 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.223256    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.224282 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.224262    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:48:01.224569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.224548    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zqxjv\""
Sep 16 12:48:01.224771 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.224548    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:48:01.224923 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.224905    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:48:01.224995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.224550    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:48:01.225223 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.225206    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8ns8b\""
Sep 16 12:48:01.225466 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.225432    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:01.225938 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.225905    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.226617 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.226598    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:48:01.226968 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.226949    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vh2jm\""
Sep 16 12:48:01.227759 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.227739    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.229134 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.228877    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:48:01.229134 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.228890    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:48:01.229134 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.228969    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kgdtv\""
Sep 16 12:48:01.229371 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.229205    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:48:01.230682 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.230622    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:48:01.230802 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.230793    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:48:01.230901 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.230843    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:48:01.231144 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.231125    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:48:01.231231 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.231125    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:48:01.232077 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-registration-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.232178 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232090    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqpq\" (UniqueName: \"kubernetes.io/projected/34788117-13c2-4d0c-a5b5-a194d52b8a28-kube-api-access-jgqpq\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.232178 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-sysconfig\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.232178 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232128    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:48:01.232178 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4b5233-e0d6-49de-a7ba-d46822a74b29-system-cni-dir\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.232178 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d4b5233-e0d6-49de-a7ba-d46822a74b29-cnibin\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.232445 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232178    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v6ctx\""
Sep 16 12:48:01.232445 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232194    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d4b5233-e0d6-49de-a7ba-d46822a74b29-cni-binary-copy\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.232445 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232248    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-etc-kubernetes\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.232445 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c444748b-7409-475d-98b9-5f14b96a1152-tmp\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.232445 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrgd\" (UniqueName: \"kubernetes.io/projected/a61a032b-e590-47e4-b7cd-e73d1ffcf9bb-kube-api-access-bbrgd\") pod \"node-ca-dw7rd\" (UID: \"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb\") " pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.232445 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-962js\" (UniqueName: \"kubernetes.io/projected/f60d9fd9-2504-4b8d-b9a6-257b20027e36-kube-api-access-962js\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:01.232445 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d4b5233-e0d6-49de-a7ba-d46822a74b29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.232445 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-cnibin\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-socket-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8d68027-95ce-41a9-ac27-37fb01f9762c-agent-certs\") pod \"konnectivity-agent-r5dpw\" (UID: \"b8d68027-95ce-41a9-ac27-37fb01f9762c\") " pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-systemd\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91daa16b-8a95-4961-a53d-81125b48ceaa-kubelet-config\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a61a032b-e590-47e4-b7cd-e73d1ffcf9bb-host\") pod \"node-ca-dw7rd\" (UID: \"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb\") " pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-run\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232673    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j966t\" (UniqueName: \"kubernetes.io/projected/8d4b5233-e0d6-49de-a7ba-d46822a74b29-kube-api-access-j966t\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232697    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-run-netns\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n97jn\" (UniqueName: \"kubernetes.io/projected/c444748b-7409-475d-98b9-5f14b96a1152-kube-api-access-n97jn\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30264cd8-3720-4fbe-8ed7-cf33859e997b-tmp-dir\") pod \"node-resolver-4p7dg\" (UID: \"30264cd8-3720-4fbe-8ed7-cf33859e997b\") " pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-host\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232824    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-lib-modules\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.232845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8d68027-95ce-41a9-ac27-37fb01f9762c-konnectivity-ca\") pod \"konnectivity-agent-r5dpw\" (UID: \"b8d68027-95ce-41a9-ac27-37fb01f9762c\") " pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-kubernetes\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232897    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-sysctl-conf\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c444748b-7409-475d-98b9-5f14b96a1152-etc-tuned\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-socket-dir-parent\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-var-lib-cni-bin\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232971    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-conf-dir\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.232986    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf5dl\" (UniqueName: \"kubernetes.io/projected/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-kube-api-access-pf5dl\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-sys-fs\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-sys\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233040    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a61a032b-e590-47e4-b7cd-e73d1ffcf9bb-serviceca\") pod \"node-ca-dw7rd\" (UID: \"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb\") " pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-run-k8s-cni-cncf-io\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-sysctl-d\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233195    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-system-cni-dir\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-device-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-modprobe-d\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.233569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233307    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-var-lib-kubelet\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-var-lib-kubelet\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-daemon-config\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-run-multus-certs\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233511    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233538    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d4b5233-e0d6-49de-a7ba-d46822a74b29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-hostroot\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-cni-binary-copy\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233621    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91daa16b-8a95-4961-a53d-81125b48ceaa-dbus\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233685    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30264cd8-3720-4fbe-8ed7-cf33859e997b-hosts-file\") pod \"node-resolver-4p7dg\" (UID: \"30264cd8-3720-4fbe-8ed7-cf33859e997b\") " pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chr8b\" (UniqueName: \"kubernetes.io/projected/30264cd8-3720-4fbe-8ed7-cf33859e997b-kube-api-access-chr8b\") pod \"node-resolver-4p7dg\" (UID: \"30264cd8-3720-4fbe-8ed7-cf33859e997b\") " pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-var-lib-cni-multus\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d4b5233-e0d6-49de-a7ba-d46822a74b29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.234469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.233749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-cni-dir\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.241252 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.241232    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:48:01.265014 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.264991    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sdvn4"
Sep 16 12:48:01.273269 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.273247    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sdvn4"
Sep 16 12:48:01.321684 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.321657    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:48:01.327265 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.327213    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal" event={"ID":"377f3cc72223911dbae51e9c32e9bb13","Type":"ContainerStarted","Data":"05806f19e41e43c66f579cede7f726b6c95ae8325072af5ec4ba9260a8a6fb2f"}
Sep 16 12:48:01.328645 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.328622    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal" event={"ID":"b59d531774340d51b7b4e079c1cee170","Type":"ContainerStarted","Data":"9c3049778e7d854de6ab9ce676534ec80c4f5da9c80ae52f95c2ce5c803743b4"}
Sep 16 12:48:01.334396 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-lib-modules\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.334518 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8d68027-95ce-41a9-ac27-37fb01f9762c-konnectivity-ca\") pod \"konnectivity-agent-r5dpw\" (UID: \"b8d68027-95ce-41a9-ac27-37fb01f9762c\") " pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:01.334518 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-kubernetes\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.334621 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-sysctl-conf\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.334621 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c444748b-7409-475d-98b9-5f14b96a1152-etc-tuned\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.334621 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334599    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-systemd-units\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.334808 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334626    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-ovnkube-config\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.334808 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-ovn-node-metrics-cert\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.334808 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-socket-dir-parent\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.334808 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-var-lib-cni-bin\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.334995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-conf-dir\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.334995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf5dl\" (UniqueName: \"kubernetes.io/projected/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-kube-api-access-pf5dl\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.334995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-sys-fs\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.334995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-sys\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.334995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.334995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.334986    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a61a032b-e590-47e4-b7cd-e73d1ffcf9bb-serviceca\") pod \"node-ca-dw7rd\" (UID: \"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb\") " pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.335275 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-run-k8s-cni-cncf-io\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.335275 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-sysctl-d\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.335275 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-system-cni-dir\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.335275 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-device-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.335275 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-modprobe-d\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.335275 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-var-lib-kubelet\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.335275 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335259    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-log-socket\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.335606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335293    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-var-lib-kubelet\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.335606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-daemon-config\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.335606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-run-multus-certs\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.335606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-slash\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.335606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:01.335606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335493    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:01.335606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d4b5233-e0d6-49de-a7ba-d46822a74b29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.335606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-hostroot\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.335606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335591    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-etc-openvswitch\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335631    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-cni-binary-copy\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91daa16b-8a95-4961-a53d-81125b48ceaa-dbus\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30264cd8-3720-4fbe-8ed7-cf33859e997b-hosts-file\") pod \"node-resolver-4p7dg\" (UID: \"30264cd8-3720-4fbe-8ed7-cf33859e997b\") " pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chr8b\" (UniqueName: \"kubernetes.io/projected/30264cd8-3720-4fbe-8ed7-cf33859e997b-kube-api-access-chr8b\") pod \"node-resolver-4p7dg\" (UID: \"30264cd8-3720-4fbe-8ed7-cf33859e997b\") " pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-var-lib-cni-multus\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335880    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-node-log\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d4b5233-e0d6-49de-a7ba-d46822a74b29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-cni-dir\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.335987    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-registration-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.336044 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqpq\" (UniqueName: \"kubernetes.io/projected/34788117-13c2-4d0c-a5b5-a194d52b8a28-kube-api-access-jgqpq\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.336558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-sysconfig\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.336558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336098    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7w49\" (UniqueName: \"kubernetes.io/projected/2d1fd77d-899a-496a-a376-98b0cdd14824-kube-api-access-s7w49\") pod \"iptables-alerter-lkwg9\" (UID: \"2d1fd77d-899a-496a-a376-98b0cdd14824\") " pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.336558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336137    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-var-lib-openvswitch\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.336558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-run-openvswitch\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.336558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4b5233-e0d6-49de-a7ba-d46822a74b29-system-cni-dir\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.336558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336246    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d4b5233-e0d6-49de-a7ba-d46822a74b29-cnibin\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.336558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336288    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d4b5233-e0d6-49de-a7ba-d46822a74b29-cni-binary-copy\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.336558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-etc-kubernetes\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.336558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c444748b-7409-475d-98b9-5f14b96a1152-tmp\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrgd\" (UniqueName: \"kubernetes.io/projected/a61a032b-e590-47e4-b7cd-e73d1ffcf9bb-kube-api-access-bbrgd\") pod \"node-ca-dw7rd\" (UID: \"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb\") " pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336625    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d1fd77d-899a-496a-a376-98b0cdd14824-iptables-alerter-script\") pod \"iptables-alerter-lkwg9\" (UID: \"2d1fd77d-899a-496a-a376-98b0cdd14824\") " pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-962js\" (UniqueName: \"kubernetes.io/projected/f60d9fd9-2504-4b8d-b9a6-257b20027e36-kube-api-access-962js\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336740    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d4b5233-e0d6-49de-a7ba-d46822a74b29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-cnibin\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336858    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-socket-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.336876    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8d68027-95ce-41a9-ac27-37fb01f9762c-agent-certs\") pod \"konnectivity-agent-r5dpw\" (UID: \"b8d68027-95ce-41a9-ac27-37fb01f9762c\") " pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336936    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-systemd\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.336968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-run-ovn\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.337038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.337549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91daa16b-8a95-4961-a53d-81125b48ceaa-kubelet-config\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.337549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a61a032b-e590-47e4-b7cd-e73d1ffcf9bb-host\") pod \"node-ca-dw7rd\" (UID: \"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb\") " pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.337549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.337549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-run\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.337549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337215    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-cni-netd\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.337549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337285    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-lib-modules\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.337549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.337549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-env-overrides\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.337549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.337540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75cm\" (UniqueName: \"kubernetes.io/projected/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-kube-api-access-t75cm\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.338450 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.338431    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs podName:f60d9fd9-2504-4b8d-b9a6-257b20027e36 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:01.838346757 +0000 UTC m=+3.019489394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs") pod "network-metrics-daemon-6xf6c" (UID: "f60d9fd9-2504-4b8d-b9a6-257b20027e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:01.338634 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.338603    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-var-lib-kubelet\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.338720 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.338629    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j966t\" (UniqueName: \"kubernetes.io/projected/8d4b5233-e0d6-49de-a7ba-d46822a74b29-kube-api-access-j966t\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.338720 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.338680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-run-netns\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.338809 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.338689    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-run-multus-certs\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.338809 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.338801    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-run-k8s-cni-cncf-io\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.338897 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.338738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n97jn\" (UniqueName: \"kubernetes.io/projected/c444748b-7409-475d-98b9-5f14b96a1152-kube-api-access-n97jn\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.338949 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.338895    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91daa16b-8a95-4961-a53d-81125b48ceaa-kubelet-config\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.339004 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.338981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-var-lib-kubelet\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.339136 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339117    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-hostroot\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.339364 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-run\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.339438 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339036    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-run-netns\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.339438 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339411    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8d68027-95ce-41a9-ac27-37fb01f9762c-konnectivity-ca\") pod \"konnectivity-agent-r5dpw\" (UID: \"b8d68027-95ce-41a9-ac27-37fb01f9762c\") " pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:01.339438 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8d4b5233-e0d6-49de-a7ba-d46822a74b29-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.339567 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339457    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-run-systemd\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.339567 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30264cd8-3720-4fbe-8ed7-cf33859e997b-tmp-dir\") pod \"node-resolver-4p7dg\" (UID: \"30264cd8-3720-4fbe-8ed7-cf33859e997b\") " pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.339567 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-host\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.339728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339553    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d1fd77d-899a-496a-a376-98b0cdd14824-host-slash\") pod \"iptables-alerter-lkwg9\" (UID: \"2d1fd77d-899a-496a-a376-98b0cdd14824\") " pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.339728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-kubelet\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.339728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339641    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-cni-bin\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.339728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-run-netns\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.339935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339707    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-cni-binary-copy\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.339935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339733    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-daemon-config\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.339935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339561    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-kubernetes\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.339935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-ovnkube-script-lib\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.339935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30264cd8-3720-4fbe-8ed7-cf33859e997b-tmp-dir\") pod \"node-resolver-4p7dg\" (UID: \"30264cd8-3720-4fbe-8ed7-cf33859e997b\") " pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.339935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339910    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-sys\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.339935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339885    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30264cd8-3720-4fbe-8ed7-cf33859e997b-hosts-file\") pod \"node-resolver-4p7dg\" (UID: \"30264cd8-3720-4fbe-8ed7-cf33859e997b\") " pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.339935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339910    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-sysctl-conf\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.340273 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339962    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4b5233-e0d6-49de-a7ba-d46822a74b29-system-cni-dir\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.340273 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339993    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-socket-dir-parent\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.340273 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.340051    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-host\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.340273 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.340113    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-var-lib-cni-bin\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.340273 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.340197    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d4b5233-e0d6-49de-a7ba-d46822a74b29-cnibin\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.340464 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.340289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-sys-fs\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.340464 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.340286    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:48:01.340464 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.340379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.340464 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.340417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-etc-kubernetes\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.340756 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.339910    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.340831 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.340779    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-conf-dir\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.340955 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.340933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91daa16b-8a95-4961-a53d-81125b48ceaa-dbus\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.341080 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341011    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d4b5233-e0d6-49de-a7ba-d46822a74b29-cni-binary-copy\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.341151 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341091    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-system-cni-dir\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.341151 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341093    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-sysctl-d\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.341151 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341138    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-modprobe-d\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.341151 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341143    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-device-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.341370 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341208    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-sysconfig\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.341370 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341212    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a61a032b-e590-47e4-b7cd-e73d1ffcf9bb-host\") pod \"node-ca-dw7rd\" (UID: \"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb\") " pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.341370 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341270    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c444748b-7409-475d-98b9-5f14b96a1152-etc-systemd\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.341370 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341275    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-multus-cni-dir\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.341370 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341324    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-socket-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.341370 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341341    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34788117-13c2-4d0c-a5b5-a194d52b8a28-registration-dir\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.341757 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.341443    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:01.341757 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341474    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d4b5233-e0d6-49de-a7ba-d46822a74b29-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.341757 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.341510    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret podName:91daa16b-8a95-4961-a53d-81125b48ceaa nodeName:}" failed. No retries permitted until 2026-09-16 12:48:01.841492243 +0000 UTC m=+3.022634891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret") pod "global-pull-secret-syncer-ncqr5" (UID: "91daa16b-8a95-4961-a53d-81125b48ceaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:01.341757 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341540    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-cnibin\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.341757 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341635    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8d4b5233-e0d6-49de-a7ba-d46822a74b29-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.342086 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341787    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a61a032b-e590-47e4-b7cd-e73d1ffcf9bb-serviceca\") pod \"node-ca-dw7rd\" (UID: \"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb\") " pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.342086 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.341855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-host-var-lib-cni-multus\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.344384 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.344362    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c444748b-7409-475d-98b9-5f14b96a1152-tmp\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.344499 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.344412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c444748b-7409-475d-98b9-5f14b96a1152-etc-tuned\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.344640 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.344619    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8d68027-95ce-41a9-ac27-37fb01f9762c-agent-certs\") pod \"konnectivity-agent-r5dpw\" (UID: \"b8d68027-95ce-41a9-ac27-37fb01f9762c\") " pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:01.348845 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.348826    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:01.348954 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.348850    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:01.348954 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.348864    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6hcxq for pod openshift-network-diagnostics/network-check-target-l2cs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:01.348954 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.348931    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq podName:7413c4f8-aa67-4b5f-b144-4a25624ea6ce nodeName:}" failed. No retries permitted until 2026-09-16 12:48:01.84891665 +0000 UTC m=+3.030059286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6hcxq" (UniqueName: "kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq") pod "network-check-target-l2cs6" (UID: "7413c4f8-aa67-4b5f-b144-4a25624ea6ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:01.352639 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.352608    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-962js\" (UniqueName: \"kubernetes.io/projected/f60d9fd9-2504-4b8d-b9a6-257b20027e36-kube-api-access-962js\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:01.352780 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.352651    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrgd\" (UniqueName: \"kubernetes.io/projected/a61a032b-e590-47e4-b7cd-e73d1ffcf9bb-kube-api-access-bbrgd\") pod \"node-ca-dw7rd\" (UID: \"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb\") " pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.352780 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.352733    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqpq\" (UniqueName: \"kubernetes.io/projected/34788117-13c2-4d0c-a5b5-a194d52b8a28-kube-api-access-jgqpq\") pod \"aws-ebs-csi-driver-node-8dd7c\" (UID: \"34788117-13c2-4d0c-a5b5-a194d52b8a28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.353494 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.353467    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chr8b\" (UniqueName: \"kubernetes.io/projected/30264cd8-3720-4fbe-8ed7-cf33859e997b-kube-api-access-chr8b\") pod \"node-resolver-4p7dg\" (UID: \"30264cd8-3720-4fbe-8ed7-cf33859e997b\") " pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.354115 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.354087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j966t\" (UniqueName: \"kubernetes.io/projected/8d4b5233-e0d6-49de-a7ba-d46822a74b29-kube-api-access-j966t\") pod \"multus-additional-cni-plugins-vhhd8\" (UID: \"8d4b5233-e0d6-49de-a7ba-d46822a74b29\") " pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.354361 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.354330    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf5dl\" (UniqueName: \"kubernetes.io/projected/ed8824d0-c77c-4605-8df5-8dd50e22c1fc-kube-api-access-pf5dl\") pod \"multus-pwrvh\" (UID: \"ed8824d0-c77c-4605-8df5-8dd50e22c1fc\") " pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.354688 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.354636    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n97jn\" (UniqueName: \"kubernetes.io/projected/c444748b-7409-475d-98b9-5f14b96a1152-kube-api-access-n97jn\") pod \"tuned-2tm9p\" (UID: \"c444748b-7409-475d-98b9-5f14b96a1152\") " pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.420239 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.420209    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:01.441219 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-slash\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-etc-openvswitch\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-node-log\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441280    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-slash\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7w49\" (UniqueName: \"kubernetes.io/projected/2d1fd77d-899a-496a-a376-98b0cdd14824-kube-api-access-s7w49\") pod \"iptables-alerter-lkwg9\" (UID: \"2d1fd77d-899a-496a-a376-98b0cdd14824\") " pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.441356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-var-lib-openvswitch\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-run-openvswitch\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441345    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-node-log\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441364    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d1fd77d-899a-496a-a376-98b0cdd14824-iptables-alerter-script\") pod \"iptables-alerter-lkwg9\" (UID: \"2d1fd77d-899a-496a-a376-98b0cdd14824\") " pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-run-ovn\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441388    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-etc-openvswitch\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-cni-netd\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441442    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-env-overrides\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441527    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75cm\" (UniqueName: \"kubernetes.io/projected/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-kube-api-access-t75cm\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-run-systemd\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d1fd77d-899a-496a-a376-98b0cdd14824-host-slash\") pod \"iptables-alerter-lkwg9\" (UID: \"2d1fd77d-899a-496a-a376-98b0cdd14824\") " pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441585    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-cni-netd\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441629    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-kubelet\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441642    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441660    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-cni-bin\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-run-netns\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.441755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441740    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-ovnkube-script-lib\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-systemd-units\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441795    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-ovnkube-config\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-ovn-node-metrics-cert\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-log-socket\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441947    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-log-socket\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.441992    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-var-lib-openvswitch\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442016    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d1fd77d-899a-496a-a376-98b0cdd14824-iptables-alerter-script\") pod \"iptables-alerter-lkwg9\" (UID: \"2d1fd77d-899a-496a-a376-98b0cdd14824\") " pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442038    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-run-openvswitch\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442075    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d1fd77d-899a-496a-a376-98b0cdd14824-host-slash\") pod \"iptables-alerter-lkwg9\" (UID: \"2d1fd77d-899a-496a-a376-98b0cdd14824\") " pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442083    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-run-ovn\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442123    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-kubelet\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-cni-bin\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442156    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-env-overrides\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-host-run-netns\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442216    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-run-systemd\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.442615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442317    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-systemd-units\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.443280 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-ovnkube-script-lib\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.443280 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.442776    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-ovnkube-config\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.444981 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.444961    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-ovn-node-metrics-cert\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.451632 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.451566    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7w49\" (UniqueName: \"kubernetes.io/projected/2d1fd77d-899a-496a-a376-98b0cdd14824-kube-api-access-s7w49\") pod \"iptables-alerter-lkwg9\" (UID: \"2d1fd77d-899a-496a-a376-98b0cdd14824\") " pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.452207 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.452186    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75cm\" (UniqueName: \"kubernetes.io/projected/bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3-kube-api-access-t75cm\") pod \"ovnkube-node-bc9vz\" (UID: \"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.522453 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.522412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c"
Sep 16 12:48:01.533012 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.532985    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:01.543896 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.543877    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4p7dg"
Sep 16 12:48:01.549634 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.549612    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vhhd8"
Sep 16 12:48:01.556952 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.556932    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dw7rd"
Sep 16 12:48:01.563838 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.563816    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2tm9p"
Sep 16 12:48:01.572946 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.572927    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pwrvh"
Sep 16 12:48:01.581003 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.580983    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lkwg9"
Sep 16 12:48:01.587853 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.587835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:01.651997 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.651967    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:48:01.667414 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:01.667389    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30264cd8_3720_4fbe_8ed7_cf33859e997b.slice/crio-fdf9c3dd05479c68d119e9111e8d478d219c0885a07e470fbfce5cb982381133 WatchSource:0}: Error finding container fdf9c3dd05479c68d119e9111e8d478d219c0885a07e470fbfce5cb982381133: Status 404 returned error can't find the container with id fdf9c3dd05479c68d119e9111e8d478d219c0885a07e470fbfce5cb982381133
Sep 16 12:48:01.668305 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:01.668279    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc444748b_7409_475d_98b9_5f14b96a1152.slice/crio-a92e9d981ff9f7f29f48c79df6b96c2b20ceff87485240444483e4cf0776df51 WatchSource:0}: Error finding container a92e9d981ff9f7f29f48c79df6b96c2b20ceff87485240444483e4cf0776df51: Status 404 returned error can't find the container with id a92e9d981ff9f7f29f48c79df6b96c2b20ceff87485240444483e4cf0776df51
Sep 16 12:48:01.673748 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:01.673701    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61a032b_e590_47e4_b7cd_e73d1ffcf9bb.slice/crio-b6c27035f51c5cecb98b18c8685eef8537eaee2bfba1a177c784dffd8e9bc37c WatchSource:0}: Error finding container b6c27035f51c5cecb98b18c8685eef8537eaee2bfba1a177c784dffd8e9bc37c: Status 404 returned error can't find the container with id b6c27035f51c5cecb98b18c8685eef8537eaee2bfba1a177c784dffd8e9bc37c
Sep 16 12:48:01.674388 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:01.674369    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8824d0_c77c_4605_8df5_8dd50e22c1fc.slice/crio-8446b5ce8c80431570fea818daeb4557a9bc5857499fc6262370acc8721bd11a WatchSource:0}: Error finding container 8446b5ce8c80431570fea818daeb4557a9bc5857499fc6262370acc8721bd11a: Status 404 returned error can't find the container with id 8446b5ce8c80431570fea818daeb4557a9bc5857499fc6262370acc8721bd11a
Sep 16 12:48:01.675535 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:01.675486    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf12b9b8_84a2_4bc9_8f7f_5fd70524fdc3.slice/crio-f8003711c4437e1fb2a6c608fbbfb983748deb0c0af2289ef54886600317e4af WatchSource:0}: Error finding container f8003711c4437e1fb2a6c608fbbfb983748deb0c0af2289ef54886600317e4af: Status 404 returned error can't find the container with id f8003711c4437e1fb2a6c608fbbfb983748deb0c0af2289ef54886600317e4af
Sep 16 12:48:01.676740 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:01.676344    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d68027_95ce_41a9_ac27_37fb01f9762c.slice/crio-55a0ee7ae00979e777c13922d989b4b4c50ec9e7aca942a05c8d59c5e19b88e8 WatchSource:0}: Error finding container 55a0ee7ae00979e777c13922d989b4b4c50ec9e7aca942a05c8d59c5e19b88e8: Status 404 returned error can't find the container with id 55a0ee7ae00979e777c13922d989b4b4c50ec9e7aca942a05c8d59c5e19b88e8
Sep 16 12:48:01.681196 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:01.681159    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1fd77d_899a_496a_a376_98b0cdd14824.slice/crio-5e9546fa122f7b79cba42cc117976a60fddd18d2febac73a1ad705c3689f573b WatchSource:0}: Error finding container 5e9546fa122f7b79cba42cc117976a60fddd18d2febac73a1ad705c3689f573b: Status 404 returned error can't find the container with id 5e9546fa122f7b79cba42cc117976a60fddd18d2febac73a1ad705c3689f573b
Sep 16 12:48:01.845797 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.845715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:01.845797 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.845761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:01.845986 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.845890    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:01.845986 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.845893    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:01.845986 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.845958    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs podName:f60d9fd9-2504-4b8d-b9a6-257b20027e36 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:02.84593896 +0000 UTC m=+4.027081596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs") pod "network-metrics-daemon-6xf6c" (UID: "f60d9fd9-2504-4b8d-b9a6-257b20027e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:01.845986 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.845978    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret podName:91daa16b-8a95-4961-a53d-81125b48ceaa nodeName:}" failed. No retries permitted until 2026-09-16 12:48:02.845968039 +0000 UTC m=+4.027110681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret") pod "global-pull-secret-syncer-ncqr5" (UID: "91daa16b-8a95-4961-a53d-81125b48ceaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:01.947110 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:01.947075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:01.947270 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.947248    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:01.947270 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.947267    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:01.947375 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.947280    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6hcxq for pod openshift-network-diagnostics/network-check-target-l2cs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:01.947375 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:01.947326    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq podName:7413c4f8-aa67-4b5f-b144-4a25624ea6ce nodeName:}" failed. No retries permitted until 2026-09-16 12:48:02.947312895 +0000 UTC m=+4.128455528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hcxq" (UniqueName: "kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq") pod "network-check-target-l2cs6" (UID: "7413c4f8-aa67-4b5f-b144-4a25624ea6ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:02.276549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.276320    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:43:01 +0000 UTC" deadline="2028-05-17 01:44:05.44846066 +0000 UTC"
Sep 16 12:48:02.276549 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.276384    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14604h56m3.172081666s"
Sep 16 12:48:02.323967 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.323383    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:02.323967 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:02.323527    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:02.345528 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.345447    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lkwg9" event={"ID":"2d1fd77d-899a-496a-a376-98b0cdd14824","Type":"ContainerStarted","Data":"5e9546fa122f7b79cba42cc117976a60fddd18d2febac73a1ad705c3689f573b"}
Sep 16 12:48:02.351578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.351496    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c" event={"ID":"34788117-13c2-4d0c-a5b5-a194d52b8a28","Type":"ContainerStarted","Data":"3b80bc5e44ab619e23038cdcd31523943c91bc53c6e557418fc71c767357d94d"}
Sep 16 12:48:02.355422 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.355309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5dpw" event={"ID":"b8d68027-95ce-41a9-ac27-37fb01f9762c","Type":"ContainerStarted","Data":"55a0ee7ae00979e777c13922d989b4b4c50ec9e7aca942a05c8d59c5e19b88e8"}
Sep 16 12:48:02.357766 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.357678    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pwrvh" event={"ID":"ed8824d0-c77c-4605-8df5-8dd50e22c1fc","Type":"ContainerStarted","Data":"8446b5ce8c80431570fea818daeb4557a9bc5857499fc6262370acc8721bd11a"}
Sep 16 12:48:02.364076 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.363147    2575 generic.go:356] "Generic (PLEG): container finished" podID="b59d531774340d51b7b4e079c1cee170" containerID="f4d99f379ea0d35987ec3fbeecc41422fde3e7eef77f6eb043953e67c724e004" exitCode=0
Sep 16 12:48:02.364076 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.363222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal" event={"ID":"b59d531774340d51b7b4e079c1cee170","Type":"ContainerDied","Data":"f4d99f379ea0d35987ec3fbeecc41422fde3e7eef77f6eb043953e67c724e004"}
Sep 16 12:48:02.390523 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.390451    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vhhd8" event={"ID":"8d4b5233-e0d6-49de-a7ba-d46822a74b29","Type":"ContainerStarted","Data":"339926290a7dac3006b2699c2e6920b4b11eca0f2f02518117816abc10577245"}
Sep 16 12:48:02.398378 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.398349    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dw7rd" event={"ID":"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb","Type":"ContainerStarted","Data":"b6c27035f51c5cecb98b18c8685eef8537eaee2bfba1a177c784dffd8e9bc37c"}
Sep 16 12:48:02.407582 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.407554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" event={"ID":"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3","Type":"ContainerStarted","Data":"f8003711c4437e1fb2a6c608fbbfb983748deb0c0af2289ef54886600317e4af"}
Sep 16 12:48:02.413078 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.413049    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tm9p" event={"ID":"c444748b-7409-475d-98b9-5f14b96a1152","Type":"ContainerStarted","Data":"a92e9d981ff9f7f29f48c79df6b96c2b20ceff87485240444483e4cf0776df51"}
Sep 16 12:48:02.415519 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.415495    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4p7dg" event={"ID":"30264cd8-3720-4fbe-8ed7-cf33859e997b","Type":"ContainerStarted","Data":"fdf9c3dd05479c68d119e9111e8d478d219c0885a07e470fbfce5cb982381133"}
Sep 16 12:48:02.874370 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.874047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:02.874370 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:02.874325    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:02.874370 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.874362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:02.874652 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:02.874402    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret podName:91daa16b-8a95-4961-a53d-81125b48ceaa nodeName:}" failed. No retries permitted until 2026-09-16 12:48:04.874381302 +0000 UTC m=+6.055523952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret") pod "global-pull-secret-syncer-ncqr5" (UID: "91daa16b-8a95-4961-a53d-81125b48ceaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:02.874652 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:02.874474    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:02.874652 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:02.874529    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs podName:f60d9fd9-2504-4b8d-b9a6-257b20027e36 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:04.874512701 +0000 UTC m=+6.055655358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs") pod "network-metrics-daemon-6xf6c" (UID: "f60d9fd9-2504-4b8d-b9a6-257b20027e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:02.975231 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:02.975189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:02.975425 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:02.975376    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:02.975425 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:02.975396    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:02.975425 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:02.975412    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6hcxq for pod openshift-network-diagnostics/network-check-target-l2cs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:02.975597 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:02.975472    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq podName:7413c4f8-aa67-4b5f-b144-4a25624ea6ce nodeName:}" failed. No retries permitted until 2026-09-16 12:48:04.975453153 +0000 UTC m=+6.156595788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hcxq" (UniqueName: "kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq") pod "network-check-target-l2cs6" (UID: "7413c4f8-aa67-4b5f-b144-4a25624ea6ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:03.276944 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:03.276834    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:43:01 +0000 UTC" deadline="2028-07-01 10:49:45.793201024 +0000 UTC"
Sep 16 12:48:03.276944 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:03.276879    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15694h1m42.516327444s"
Sep 16 12:48:03.323409 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:03.322866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:03.323409 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:03.322999    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:03.323922 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:03.323763    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:03.323922 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:03.323870    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:03.427352 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:03.426629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal" event={"ID":"b59d531774340d51b7b4e079c1cee170","Type":"ContainerStarted","Data":"fbecac09fc41736c15daf2f2a87e90baca9c3c2ee6d7fc469f1aeac75edc7af1"}
Sep 16 12:48:03.464532 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:03.464471    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-92.ec2.internal" podStartSLOduration=2.464449246 podStartE2EDuration="2.464449246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:03.464439645 +0000 UTC m=+4.645582303" watchObservedRunningTime="2026-09-16 12:48:03.464449246 +0000 UTC m=+4.645591908"
Sep 16 12:48:04.322577 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:04.322525    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:04.323082 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:04.322707    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:04.895486 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:04.894626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:04.895486 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:04.894768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:04.895486 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:04.894899    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:04.895486 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:04.894961    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret podName:91daa16b-8a95-4961-a53d-81125b48ceaa nodeName:}" failed. No retries permitted until 2026-09-16 12:48:08.894942822 +0000 UTC m=+10.076085474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret") pod "global-pull-secret-syncer-ncqr5" (UID: "91daa16b-8a95-4961-a53d-81125b48ceaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:04.895486 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:04.895395    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:04.895486 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:04.895451    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs podName:f60d9fd9-2504-4b8d-b9a6-257b20027e36 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:08.895433217 +0000 UTC m=+10.076575853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs") pod "network-metrics-daemon-6xf6c" (UID: "f60d9fd9-2504-4b8d-b9a6-257b20027e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:04.996101 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:04.996055    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:04.996279 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:04.996259    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:04.996345 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:04.996287    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:04.996345 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:04.996302    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6hcxq for pod openshift-network-diagnostics/network-check-target-l2cs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:04.996439 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:04.996362    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq podName:7413c4f8-aa67-4b5f-b144-4a25624ea6ce nodeName:}" failed. No retries permitted until 2026-09-16 12:48:08.996342611 +0000 UTC m=+10.177485257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hcxq" (UniqueName: "kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq") pod "network-check-target-l2cs6" (UID: "7413c4f8-aa67-4b5f-b144-4a25624ea6ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:05.323883 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:05.323801    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:05.323883 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:05.323822    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:05.324389 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:05.323941    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:05.324389 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:05.324093    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:06.323143 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:06.323102    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:06.323331 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:06.323244    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:07.327649 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:07.327139    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:07.327649 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:07.327271    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:07.327649 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:07.327604    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:07.328364 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:07.327730    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:08.323274 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:08.323187    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:08.323418 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:08.323327    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:08.930425 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:08.929779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:08.930425 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:08.929839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:08.930425 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:08.930027    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:08.930425 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:08.930093    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs podName:f60d9fd9-2504-4b8d-b9a6-257b20027e36 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:16.930071831 +0000 UTC m=+18.111214468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs") pod "network-metrics-daemon-6xf6c" (UID: "f60d9fd9-2504-4b8d-b9a6-257b20027e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:08.930425 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:08.930146    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:08.930425 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:08.930201    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret podName:91daa16b-8a95-4961-a53d-81125b48ceaa nodeName:}" failed. No retries permitted until 2026-09-16 12:48:16.930183872 +0000 UTC m=+18.111326521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret") pod "global-pull-secret-syncer-ncqr5" (UID: "91daa16b-8a95-4961-a53d-81125b48ceaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:09.040154 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:09.040111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:09.041239 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:09.040581    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:09.041239 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:09.040976    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:09.041239 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:09.040999    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6hcxq for pod openshift-network-diagnostics/network-check-target-l2cs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:09.041239 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:09.041066    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq podName:7413c4f8-aa67-4b5f-b144-4a25624ea6ce nodeName:}" failed. No retries permitted until 2026-09-16 12:48:17.041045653 +0000 UTC m=+18.222188289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hcxq" (UniqueName: "kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq") pod "network-check-target-l2cs6" (UID: "7413c4f8-aa67-4b5f-b144-4a25624ea6ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:09.324345 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:09.324258    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:09.324528 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:09.324412    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:09.324528 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:09.324459    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:09.324651 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:09.324549    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:10.322963 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:10.322927    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:10.323470 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:10.323068    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:11.323790 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:11.323735    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:11.324211 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:11.323868    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:11.325689 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:11.324275    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:11.325689 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:11.324360    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:12.323533 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:12.323498    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:12.323771 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:12.323634    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:13.322590 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:13.322556    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:13.323055 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:13.322703    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:13.323055 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:13.322556    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:13.323055 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:13.322898    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:14.323380 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:14.322887    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:14.323380 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:14.323028    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:15.323547 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:15.323505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:15.323924 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:15.323505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:15.323924 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:15.323656    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:15.323924 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:15.323729    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:16.322789 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:16.322754    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:16.323008 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:16.322883    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:16.998322 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:16.998281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:16.998828 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:16.998337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:16.998828 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:16.998445    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:16.998828 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:16.998445    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:16.998828 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:16.998503    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs podName:f60d9fd9-2504-4b8d-b9a6-257b20027e36 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:32.998484319 +0000 UTC m=+34.179626965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs") pod "network-metrics-daemon-6xf6c" (UID: "f60d9fd9-2504-4b8d-b9a6-257b20027e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:16.998828 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:16.998527    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret podName:91daa16b-8a95-4961-a53d-81125b48ceaa nodeName:}" failed. No retries permitted until 2026-09-16 12:48:32.998518508 +0000 UTC m=+34.179661138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret") pod "global-pull-secret-syncer-ncqr5" (UID: "91daa16b-8a95-4961-a53d-81125b48ceaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:17.098669 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:17.098614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:17.098819 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:17.098776    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:17.098819 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:17.098799    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:17.098819 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:17.098809    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6hcxq for pod openshift-network-diagnostics/network-check-target-l2cs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:17.098945 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:17.098861    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq podName:7413c4f8-aa67-4b5f-b144-4a25624ea6ce nodeName:}" failed. No retries permitted until 2026-09-16 12:48:33.098846619 +0000 UTC m=+34.279989254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hcxq" (UniqueName: "kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq") pod "network-check-target-l2cs6" (UID: "7413c4f8-aa67-4b5f-b144-4a25624ea6ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:17.322803 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:17.322735    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:17.322945 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:17.322737    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:17.322945 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:17.322840    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:17.322945 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:17.322936    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:18.323365 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:18.323334    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:18.323836 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:18.323468    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:19.324165 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:19.324124    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:19.324514 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:19.324222    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:19.324514 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:19.324309    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:19.324514 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:19.324398    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:20.323099 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:20.323066    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:20.323254 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:20.323181    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:21.322797 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.322610    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:21.323573 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:21.322845    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:21.323573 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.322706    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:21.323573 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:21.322911    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:21.469111 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.469081    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d4b5233-e0d6-49de-a7ba-d46822a74b29" containerID="bcedcc409c1d233249b493d5b1b6b252717ac8ffe4f2d54df84c8c505dfaaf15" exitCode=0
Sep 16 12:48:21.469232 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.469162    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vhhd8" event={"ID":"8d4b5233-e0d6-49de-a7ba-d46822a74b29","Type":"ContainerDied","Data":"bcedcc409c1d233249b493d5b1b6b252717ac8ffe4f2d54df84c8c505dfaaf15"}
Sep 16 12:48:21.470815 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.470789    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dw7rd" event={"ID":"a61a032b-e590-47e4-b7cd-e73d1ffcf9bb","Type":"ContainerStarted","Data":"ae46706ae05ff74b7ff1528d038b109465d20e1c429bd81935e7e1ee6fea01db"}
Sep 16 12:48:21.473314 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.473292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" event={"ID":"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3","Type":"ContainerStarted","Data":"52c81c02338bc0ff83db9f40aff5ea6daa8690d47c7b1760d73429f6dd5b5a36"}
Sep 16 12:48:21.473413 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.473368    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" event={"ID":"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3","Type":"ContainerStarted","Data":"f46059fc3b52e8ba7241de1c9b8b652fec75a324cd72962a6189cfeff6350bd8"}
Sep 16 12:48:21.473476 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.473417    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" event={"ID":"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3","Type":"ContainerStarted","Data":"d44f0abadd79920a80537c81054b97021e1e50cae4c86ee4049b80e5387b8a71"}
Sep 16 12:48:21.473476 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.473467    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" event={"ID":"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3","Type":"ContainerStarted","Data":"fc2ddd852e083a09c565fef0fcc06dc7219dfb251685e41c0fef0b09b4acbcb4"}
Sep 16 12:48:21.474799 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.474773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2tm9p" event={"ID":"c444748b-7409-475d-98b9-5f14b96a1152","Type":"ContainerStarted","Data":"63eb5beb2178338be55e3222ca53853a3040d01ea6ceae7936772e0e684caa61"}
Sep 16 12:48:21.476141 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.476123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4p7dg" event={"ID":"30264cd8-3720-4fbe-8ed7-cf33859e997b","Type":"ContainerStarted","Data":"8c8d53ed85c360aae4fc3e6cd3ce4b6628bf81fb9645cf3e2c180795d70bc799"}
Sep 16 12:48:21.477419 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.477399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal" event={"ID":"377f3cc72223911dbae51e9c32e9bb13","Type":"ContainerStarted","Data":"a43f519ceaa5e24ddf70032dad5159ce6ed67f91c698e4de1a01b160911ed2ca"}
Sep 16 12:48:21.478829 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.478800    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c" event={"ID":"34788117-13c2-4d0c-a5b5-a194d52b8a28","Type":"ContainerStarted","Data":"795cdf688d13127235e567dfb169c6266497c88e94cbb614f828a4102dd3d9b8"}
Sep 16 12:48:21.480405 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.480380    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5dpw" event={"ID":"b8d68027-95ce-41a9-ac27-37fb01f9762c","Type":"ContainerStarted","Data":"f4ca99ca781f53317188281db9b81061c1426859b09188788b5b2c4bec9ced59"}
Sep 16 12:48:21.481763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.481744    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pwrvh" event={"ID":"ed8824d0-c77c-4605-8df5-8dd50e22c1fc","Type":"ContainerStarted","Data":"c32e39f9bfad1c84878c1ced69a627b517b14ea2766fb69ec02db3d5fd36eab7"}
Sep 16 12:48:21.511437 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.511402    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r5dpw" podStartSLOduration=2.164211903 podStartE2EDuration="21.511388895s" totalImagesPullingTime="19.347176992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:21.511188916 +0000 UTC m=+22.692331568" watchObservedRunningTime="2026-09-16 12:48:21.511388895 +0000 UTC m=+22.692531548"
Sep 16 12:48:21.526716 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.526683    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4p7dg" podStartSLOduration=2.169436595 podStartE2EDuration="21.526666108s" totalImagesPullingTime="19.357229513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:21.526205496 +0000 UTC m=+22.707348150" watchObservedRunningTime="2026-09-16 12:48:21.526666108 +0000 UTC m=+22.707808752"
Sep 16 12:48:21.555996 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.555823    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-92.ec2.internal" podStartSLOduration=20.555811455 podStartE2EDuration="20.555811455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:21.555709258 +0000 UTC m=+22.736851913" watchObservedRunningTime="2026-09-16 12:48:21.555811455 +0000 UTC m=+22.736954106"
Sep 16 12:48:21.556435 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.556405    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dw7rd" podStartSLOduration=2.206526979 podStartE2EDuration="21.556397514s" totalImagesPullingTime="19.349870535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:21.542097919 +0000 UTC m=+22.723240573" watchObservedRunningTime="2026-09-16 12:48:21.556397514 +0000 UTC m=+22.737540170"
Sep 16 12:48:21.591079 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.591041    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2tm9p" podStartSLOduration=2.204574757 podStartE2EDuration="21.591029711s" totalImagesPullingTime="19.386454954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:21.573120689 +0000 UTC m=+22.754263341" watchObservedRunningTime="2026-09-16 12:48:21.591029711 +0000 UTC m=+22.772172363"
Sep 16 12:48:21.591240 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:21.591215    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pwrvh" podStartSLOduration=2.186699769 podStartE2EDuration="21.59120795s" totalImagesPullingTime="19.404508181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:21.590747817 +0000 UTC m=+22.771890471" watchObservedRunningTime="2026-09-16 12:48:21.59120795 +0000 UTC m=+22.772350602"
Sep 16 12:48:22.322888 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:22.322820    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:22.323305 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:22.322942    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:22.392255 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:22.392232    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:48:22.490372 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:22.490337    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" event={"ID":"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3","Type":"ContainerStarted","Data":"b2a33799a24e8fe1cf371e573731b7893ea08ce981e3521b029bb041da8e2378"}
Sep 16 12:48:22.490568 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:22.490442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" event={"ID":"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3","Type":"ContainerStarted","Data":"eabc9bb581cca7837ac46705bd0ac625bb4019c8e05f9d5c16ecaa86dcd1af18"}
Sep 16 12:48:22.492248 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:22.492217    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lkwg9" event={"ID":"2d1fd77d-899a-496a-a376-98b0cdd14824","Type":"ContainerStarted","Data":"a674068dba5bc6e8e7fe8222568f501bc6acdd1398242ca333c8e3c41157a510"}
Sep 16 12:48:22.494264 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:22.494235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c" event={"ID":"34788117-13c2-4d0c-a5b5-a194d52b8a28","Type":"ContainerStarted","Data":"4104e377f9a91ae192cb356ad11c5d4d76bc263f4b3a2e10f00dfcf0f69646d4"}
Sep 16 12:48:22.509434 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:22.509387    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lkwg9" podStartSLOduration=3.168470248 podStartE2EDuration="22.509370677s" totalImagesPullingTime="19.340900429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:22.50887065 +0000 UTC m=+23.690013304" watchObservedRunningTime="2026-09-16 12:48:22.509370677 +0000 UTC m=+23.690513331"
Sep 16 12:48:23.292969 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:23.292860    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:48:22.392249088Z","UUID":"b3488e26-d661-4e94-a2e9-42003cb08691","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:48:23.294545 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:23.294523    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:48:23.294737 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:23.294556    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:48:23.322664 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:23.322623    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:23.322810 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:23.322623    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:23.322810 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:23.322768    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:23.322923 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:23.322827    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:24.322831 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:24.322755    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:24.323046 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:24.322888    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:24.348351 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:24.348324    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:24.349027 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:24.349001    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:24.502922 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:24.502888    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" event={"ID":"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3","Type":"ContainerStarted","Data":"208c0277ec2198c82ebc551126a8b7348109ed8ec5018e10bbf4c48ce0b6a824"}
Sep 16 12:48:24.505212 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:24.505185    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c" event={"ID":"34788117-13c2-4d0c-a5b5-a194d52b8a28","Type":"ContainerStarted","Data":"d9163c7b17e4e9997534da18c73f2b813deee5eca4de887155f83b40a781b218"}
Sep 16 12:48:24.528623 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:24.528572    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dd7c" podStartSLOduration=2.963617765 podStartE2EDuration="24.528554438s" totalImagesPullingTime="21.564936673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:24.527816451 +0000 UTC m=+25.708959104" watchObservedRunningTime="2026-09-16 12:48:24.528554438 +0000 UTC m=+25.709697095"
Sep 16 12:48:25.323391 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:25.323360    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:25.323391 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:25.323382    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:25.323838 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:25.323474    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:25.323838 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:25.323613    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:26.323604 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:26.323432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:26.324169 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:26.323737    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:26.511093 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:26.511023    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d4b5233-e0d6-49de-a7ba-d46822a74b29" containerID="c89756cccf218c13f0ab30cfc97361ca0d1becd46e45032cfd1a57c66b514a2a" exitCode=0
Sep 16 12:48:26.511093 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:26.511067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vhhd8" event={"ID":"8d4b5233-e0d6-49de-a7ba-d46822a74b29","Type":"ContainerDied","Data":"c89756cccf218c13f0ab30cfc97361ca0d1becd46e45032cfd1a57c66b514a2a"}
Sep 16 12:48:26.514938 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:26.514919    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" event={"ID":"bf12b9b8-84a2-4bc9-8f7f-5fd70524fdc3","Type":"ContainerStarted","Data":"d46c549df5b9d9406815baeeb1e87344cf65df0a8213a22c9470d8998ebef2d4"}
Sep 16 12:48:26.515220 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:26.515195    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:26.515308 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:26.515257    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:26.530934 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:26.530915    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:26.575319 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:26.575278    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz" podStartSLOduration=7.152475887 podStartE2EDuration="26.575265507s" totalImagesPullingTime="19.42278962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:26.57479814 +0000 UTC m=+27.755940793" watchObservedRunningTime="2026-09-16 12:48:26.575265507 +0000 UTC m=+27.756408443"
Sep 16 12:48:27.323684 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:27.323511    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:27.323684 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:27.323630    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:27.324558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:27.323698    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:27.324558 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:27.323797    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:27.434877 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:27.434847    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:27.517777 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:27.517750    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:27.533479 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:27.533453    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:27.543575 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:27.543552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:28.081117 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:28.080881    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xf6c"]
Sep 16 12:48:28.081318 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:28.081172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:28.081318 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:28.081292    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:28.088919 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:28.088884    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncqr5"]
Sep 16 12:48:28.089041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:28.088980    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:28.089097 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:28.089076    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:28.089472 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:28.089454    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l2cs6"]
Sep 16 12:48:28.089564 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:28.089550    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:28.089684 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:28.089639    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:28.521518 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:28.521441    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d4b5233-e0d6-49de-a7ba-d46822a74b29" containerID="4700aeae18c9ee159ef22420f70233d388a2a85ebba6076d7e98422689a231af" exitCode=0
Sep 16 12:48:28.521932 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:28.521513    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vhhd8" event={"ID":"8d4b5233-e0d6-49de-a7ba-d46822a74b29","Type":"ContainerDied","Data":"4700aeae18c9ee159ef22420f70233d388a2a85ebba6076d7e98422689a231af"}
Sep 16 12:48:29.225171 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:29.225144    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:29.324678 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:29.324634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:29.452339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:29.452311    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:29.452498 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:29.452425    2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 12:48:29.452944 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:29.452924    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r5dpw"
Sep 16 12:48:29.531367 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:29.531313    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:30.323006 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:30.322979    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:30.323143 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:30.322979    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:30.323143 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:30.323063    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:30.323143 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:30.323078    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:30.323236 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:30.323150    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:30.323236 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:30.323217    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:30.528253 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:30.528174    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d4b5233-e0d6-49de-a7ba-d46822a74b29" containerID="cc442e7a4e5a06e91df20d2237137a602bcd636bdba072d4791dd9f1d9e0e47a" exitCode=0
Sep 16 12:48:30.528253 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:30.528230    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vhhd8" event={"ID":"8d4b5233-e0d6-49de-a7ba-d46822a74b29","Type":"ContainerDied","Data":"cc442e7a4e5a06e91df20d2237137a602bcd636bdba072d4791dd9f1d9e0e47a"}
Sep 16 12:48:32.323188 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:32.322777    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:32.323188 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:32.322826    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:32.323188 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:32.322885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:32.323188 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:32.322902    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:32.323188 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:32.322950    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:32.323188 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:32.323044    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:32.691639 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:32.691580    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:33.015660 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:33.015614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:33.015829 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:33.015679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:33.015829 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:33.015761    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:33.015829 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:33.015787    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:33.015829 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:33.015826    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs podName:f60d9fd9-2504-4b8d-b9a6-257b20027e36 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:05.015807981 +0000 UTC m=+66.196950611 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs") pod "network-metrics-daemon-6xf6c" (UID: "f60d9fd9-2504-4b8d-b9a6-257b20027e36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:48:33.015996 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:33.015841    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret podName:91daa16b-8a95-4961-a53d-81125b48ceaa nodeName:}" failed. No retries permitted until 2026-09-16 12:49:05.015833249 +0000 UTC m=+66.196975881 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret") pod "global-pull-secret-syncer-ncqr5" (UID: "91daa16b-8a95-4961-a53d-81125b48ceaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:48:33.117071 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:33.116869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:33.117245 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:33.117032    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:48:33.117245 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:33.117144    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:48:33.117245 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:33.117160    2575 projected.go:196] Error preparing data for projected volume kube-api-access-6hcxq for pod openshift-network-diagnostics/network-check-target-l2cs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:33.117245 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:33.117219    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq podName:7413c4f8-aa67-4b5f-b144-4a25624ea6ce nodeName:}" failed. No retries permitted until 2026-09-16 12:49:05.117204353 +0000 UTC m=+66.298346985 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hcxq" (UniqueName: "kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq") pod "network-check-target-l2cs6" (UID: "7413c4f8-aa67-4b5f-b144-4a25624ea6ce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:48:34.244325 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:34.244289    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:34.322901 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:34.322876    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:34.323066 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:34.322876    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:34.323066 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:34.322991    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncqr5" podUID="91daa16b-8a95-4961-a53d-81125b48ceaa"
Sep 16 12:48:34.323066 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:34.322877    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:34.323234 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:34.323072    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l2cs6" podUID="7413c4f8-aa67-4b5f-b144-4a25624ea6ce"
Sep 16 12:48:34.323234 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:34.323152    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xf6c" podUID="f60d9fd9-2504-4b8d-b9a6-257b20027e36"
Sep 16 12:48:34.884149 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:34.884108    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:34.981638 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:34.981615    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:35.062950 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.062929    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-92.ec2.internal" event="NodeReady"
Sep 16 12:48:35.063071 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.063025    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:48:35.124144 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.124119    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wfs82"]
Sep 16 12:48:35.125924 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.125909    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.129785 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.129764    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kr5tj"]
Sep 16 12:48:35.131186 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.131166    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rwp6v\""
Sep 16 12:48:35.131280 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.131192    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:48:35.131280 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.131166    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:48:35.131699 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.131680    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kr5tj"
Sep 16 12:48:35.136002 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.135958    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:48:35.136493 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.136478    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:48:35.136493 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.136487    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:48:35.137437 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.137424    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t2wwc\""
Sep 16 12:48:35.148834 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.148776    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wfs82"]
Sep 16 12:48:35.149399 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.149382    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kr5tj"]
Sep 16 12:48:35.232187 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.232164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7n2h\" (UniqueName: \"kubernetes.io/projected/497b9dd9-4306-4db6-92de-958ee5067c9a-kube-api-access-m7n2h\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.232309 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.232205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/497b9dd9-4306-4db6-92de-958ee5067c9a-config-volume\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.232309 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.232254    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/497b9dd9-4306-4db6-92de-958ee5067c9a-tmp-dir\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.232309 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.232276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmlh\" (UniqueName: \"kubernetes.io/projected/0db4d710-77d0-4884-a702-739868c1c6ad-kube-api-access-lgmlh\") pod \"ingress-canary-kr5tj\" (UID: \"0db4d710-77d0-4884-a702-739868c1c6ad\") " pod="openshift-ingress-canary/ingress-canary-kr5tj"
Sep 16 12:48:35.232448 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.232321    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.232448 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.232350    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0db4d710-77d0-4884-a702-739868c1c6ad-cert\") pod \"ingress-canary-kr5tj\" (UID: \"0db4d710-77d0-4884-a702-739868c1c6ad\") " pod="openshift-ingress-canary/ingress-canary-kr5tj"
Sep 16 12:48:35.283837 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.283808    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:35.333261 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.333235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7n2h\" (UniqueName: \"kubernetes.io/projected/497b9dd9-4306-4db6-92de-958ee5067c9a-kube-api-access-m7n2h\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.333363 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.333274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/497b9dd9-4306-4db6-92de-958ee5067c9a-config-volume\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.333363 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.333314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/497b9dd9-4306-4db6-92de-958ee5067c9a-tmp-dir\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.333363 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.333339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmlh\" (UniqueName: \"kubernetes.io/projected/0db4d710-77d0-4884-a702-739868c1c6ad-kube-api-access-lgmlh\") pod \"ingress-canary-kr5tj\" (UID: \"0db4d710-77d0-4884-a702-739868c1c6ad\") " pod="openshift-ingress-canary/ingress-canary-kr5tj"
Sep 16 12:48:35.333524 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.333385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.333524 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.333415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0db4d710-77d0-4884-a702-739868c1c6ad-cert\") pod \"ingress-canary-kr5tj\" (UID: \"0db4d710-77d0-4884-a702-739868c1c6ad\") " pod="openshift-ingress-canary/ingress-canary-kr5tj"
Sep 16 12:48:35.333524 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:35.333496    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:35.333689 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:35.333559    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls podName:497b9dd9-4306-4db6-92de-958ee5067c9a nodeName:}" failed. No retries permitted until 2026-09-16 12:48:35.833544409 +0000 UTC m=+37.014687040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls") pod "dns-default-wfs82" (UID: "497b9dd9-4306-4db6-92de-958ee5067c9a") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:35.333749 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.333710    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/497b9dd9-4306-4db6-92de-958ee5067c9a-tmp-dir\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.333868 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.333851    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/497b9dd9-4306-4db6-92de-958ee5067c9a-config-volume\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.337524 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.337504    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0db4d710-77d0-4884-a702-739868c1c6ad-cert\") pod \"ingress-canary-kr5tj\" (UID: \"0db4d710-77d0-4884-a702-739868c1c6ad\") " pod="openshift-ingress-canary/ingress-canary-kr5tj"
Sep 16 12:48:35.342849 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.342831    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7n2h\" (UniqueName: \"kubernetes.io/projected/497b9dd9-4306-4db6-92de-958ee5067c9a-kube-api-access-m7n2h\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.343795 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.343777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmlh\" (UniqueName: \"kubernetes.io/projected/0db4d710-77d0-4884-a702-739868c1c6ad-kube-api-access-lgmlh\") pod \"ingress-canary-kr5tj\" (UID: \"0db4d710-77d0-4884-a702-739868c1c6ad\") " pod="openshift-ingress-canary/ingress-canary-kr5tj"
Sep 16 12:48:35.442005 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.441940    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kr5tj"
Sep 16 12:48:35.446367 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.446344    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:35.544224 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.544198    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d4b5233-e0d6-49de-a7ba-d46822a74b29" containerID="73a9121a87fcb7ff501328024a8be4fbffea4811e90f28575ab927cf7b360f96" exitCode=0
Sep 16 12:48:35.544347 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.544244    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vhhd8" event={"ID":"8d4b5233-e0d6-49de-a7ba-d46822a74b29","Type":"ContainerDied","Data":"73a9121a87fcb7ff501328024a8be4fbffea4811e90f28575ab927cf7b360f96"}
Sep 16 12:48:35.728884 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.728832    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kr5tj"]
Sep 16 12:48:35.732510 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:48:35.732485    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db4d710_77d0_4884_a702_739868c1c6ad.slice/crio-0dc4455b207ad036dcdd83ac8f4ea126f7b99754e7b5125906d1545d513d8a07 WatchSource:0}: Error finding container 0dc4455b207ad036dcdd83ac8f4ea126f7b99754e7b5125906d1545d513d8a07: Status 404 returned error can't find the container with id 0dc4455b207ad036dcdd83ac8f4ea126f7b99754e7b5125906d1545d513d8a07
Sep 16 12:48:35.837109 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.837081    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:35.837225 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:35.837205    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:35.837293 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:35.837264    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls podName:497b9dd9-4306-4db6-92de-958ee5067c9a nodeName:}" failed. No retries permitted until 2026-09-16 12:48:36.837248944 +0000 UTC m=+38.018391575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls") pod "dns-default-wfs82" (UID: "497b9dd9-4306-4db6-92de-958ee5067c9a") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:35.889621 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.889596    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:35.953832 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:35.953807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:36.323230 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.322995    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:48:36.324041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.323028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:48:36.324041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.323028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:48:36.327889 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.327870    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:48:36.346104 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.346033    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:48:36.346104 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.346067    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:48:36.346383 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.346365    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vlkh7\""
Sep 16 12:48:36.346691 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.346670    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:48:36.346795 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.346754    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v25dx\""
Sep 16 12:48:36.550533 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.550412    2575 generic.go:356] "Generic (PLEG): container finished" podID="8d4b5233-e0d6-49de-a7ba-d46822a74b29" containerID="c282b32ec540bbc43de71830d7f8a4146b340489a83bd408de7a295d564dc431" exitCode=0
Sep 16 12:48:36.550533 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.550461    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vhhd8" event={"ID":"8d4b5233-e0d6-49de-a7ba-d46822a74b29","Type":"ContainerDied","Data":"c282b32ec540bbc43de71830d7f8a4146b340489a83bd408de7a295d564dc431"}
Sep 16 12:48:36.552143 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.552002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kr5tj" event={"ID":"0db4d710-77d0-4884-a702-739868c1c6ad","Type":"ContainerStarted","Data":"0dc4455b207ad036dcdd83ac8f4ea126f7b99754e7b5125906d1545d513d8a07"}
Sep 16 12:48:36.844515 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.844478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:36.844722 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:36.844618    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:36.844722 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:36.844702    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls podName:497b9dd9-4306-4db6-92de-958ee5067c9a nodeName:}" failed. No retries permitted until 2026-09-16 12:48:38.844683341 +0000 UTC m=+40.025825992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls") pod "dns-default-wfs82" (UID: "497b9dd9-4306-4db6-92de-958ee5067c9a") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:36.950727 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:36.950700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:37.043877 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:37.043847    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:37.558865 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:37.558830    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vhhd8" event={"ID":"8d4b5233-e0d6-49de-a7ba-d46822a74b29","Type":"ContainerStarted","Data":"060e1019b9cbf48e9d5d3b02e3798538893b95413e6d98444b069a8dff54dd51"}
Sep 16 12:48:37.589259 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:37.589211    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vhhd8" podStartSLOduration=7.650433324 podStartE2EDuration="37.589196172s" totalImagesPullingTime="29.938762848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:37.587250878 +0000 UTC m=+38.768393546" watchObservedRunningTime="2026-09-16 12:48:37.589196172 +0000 UTC m=+38.770338825"
Sep 16 12:48:37.723160 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:37.723125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:37.823033 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:37.822955    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:38.480751 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:38.480705    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:38.562019 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:38.561987    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kr5tj" event={"ID":"0db4d710-77d0-4884-a702-739868c1c6ad","Type":"ContainerStarted","Data":"6fa2a74a57e3dbcfdf1c2bfc7d0588d8ab6ad59853fe7d31b36d54106d67ca18"}
Sep 16 12:48:38.583663 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:38.583612    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kr5tj" podStartSLOduration=1.09506083 podStartE2EDuration="3.58359774s" totalImagesPullingTime="2.48853691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:38.583289158 +0000 UTC m=+39.764431811" watchObservedRunningTime="2026-09-16 12:48:38.58359774 +0000 UTC m=+39.764740392"
Sep 16 12:48:38.613532 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:38.613506    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:38.860225 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:38.860202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:38.860367 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:38.860328    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:38.860411 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:38.860382    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls podName:497b9dd9-4306-4db6-92de-958ee5067c9a nodeName:}" failed. No retries permitted until 2026-09-16 12:48:42.860367313 +0000 UTC m=+44.041509944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls") pod "dns-default-wfs82" (UID: "497b9dd9-4306-4db6-92de-958ee5067c9a") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:39.802055 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:39.802022    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:42.883435 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:42.883401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:42.883780 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:42.883554    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:42.883780 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:42.883632    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls podName:497b9dd9-4306-4db6-92de-958ee5067c9a nodeName:}" failed. No retries permitted until 2026-09-16 12:48:50.883613709 +0000 UTC m=+52.064756357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls") pod "dns-default-wfs82" (UID: "497b9dd9-4306-4db6-92de-958ee5067c9a") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:47.609197 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:47.609154    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:47.769294 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:47.769261    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:50.931828 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:50.931791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:48:50.932187 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:50.931909    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:48:50.932187 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:48:50.931974    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls podName:497b9dd9-4306-4db6-92de-958ee5067c9a nodeName:}" failed. No retries permitted until 2026-09-16 12:49:06.931956042 +0000 UTC m=+68.113098677 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls") pod "dns-default-wfs82" (UID: "497b9dd9-4306-4db6-92de-958ee5067c9a") : secret "dns-default-metrics-tls" not found
Sep 16 12:48:51.052776 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:51.052750    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:51.163375 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:51.163348    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:52.288432 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:52.288400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:52.402518 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:52.402485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:53.299969 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:53.299934    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:53.447345 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:53.447305    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:59.538225 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:59.538195    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bc9vz"
Sep 16 12:48:59.720597 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:59.720568    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:48:59.810180 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:48:59.810122    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:01.963080 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:01.963046    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:05.020160 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.020107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:49:05.020160 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.020161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:49:05.026465 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.026442    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:49:05.026630 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.026609    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:49:05.031278 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:05.031261    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:49:05.031369 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:05.031327    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs podName:f60d9fd9-2504-4b8d-b9a6-257b20027e36 nodeName:}" failed. No retries permitted until 2026-09-16 12:50:09.031306198 +0000 UTC m=+130.212448835 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs") pod "network-metrics-daemon-6xf6c" (UID: "f60d9fd9-2504-4b8d-b9a6-257b20027e36") : secret "metrics-daemon-secret" not found
Sep 16 12:49:05.034418 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.034389    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91daa16b-8a95-4961-a53d-81125b48ceaa-original-pull-secret\") pod \"global-pull-secret-syncer-ncqr5\" (UID: \"91daa16b-8a95-4961-a53d-81125b48ceaa\") " pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:49:05.120753 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.120718    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:49:05.123890 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.123876    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:49:05.135493 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.135477    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncqr5"
Sep 16 12:49:05.135790 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.135773    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:49:05.144928 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.144903    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcxq\" (UniqueName: \"kubernetes.io/projected/7413c4f8-aa67-4b5f-b144-4a25624ea6ce-kube-api-access-6hcxq\") pod \"network-check-target-l2cs6\" (UID: \"7413c4f8-aa67-4b5f-b144-4a25624ea6ce\") " pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:49:05.412540 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.412513    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncqr5"]
Sep 16 12:49:05.415306 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:05.415278    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91daa16b_8a95_4961_a53d_81125b48ceaa.slice/crio-eb667eb792e094c37cefb8c408ef2825316a87b18eab926fb5c17b6079b6b402 WatchSource:0}: Error finding container eb667eb792e094c37cefb8c408ef2825316a87b18eab926fb5c17b6079b6b402: Status 404 returned error can't find the container with id eb667eb792e094c37cefb8c408ef2825316a87b18eab926fb5c17b6079b6b402
Sep 16 12:49:05.447142 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.447120    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vlkh7\""
Sep 16 12:49:05.453595 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.453579    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:49:05.620062 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.620033    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ncqr5" event={"ID":"91daa16b-8a95-4961-a53d-81125b48ceaa","Type":"ContainerStarted","Data":"eb667eb792e094c37cefb8c408ef2825316a87b18eab926fb5c17b6079b6b402"}
Sep 16 12:49:05.726836 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:05.726750    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l2cs6"]
Sep 16 12:49:05.730115 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:05.730070    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7413c4f8_aa67_4b5f_b144_4a25624ea6ce.slice/crio-9a5e4d329c2f54ad5cedccce467dbac27075f6111a25491a05bac1089a9a9d57 WatchSource:0}: Error finding container 9a5e4d329c2f54ad5cedccce467dbac27075f6111a25491a05bac1089a9a9d57: Status 404 returned error can't find the container with id 9a5e4d329c2f54ad5cedccce467dbac27075f6111a25491a05bac1089a9a9d57
Sep 16 12:49:06.624598 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:06.624555    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l2cs6" event={"ID":"7413c4f8-aa67-4b5f-b144-4a25624ea6ce","Type":"ContainerStarted","Data":"9a5e4d329c2f54ad5cedccce467dbac27075f6111a25491a05bac1089a9a9d57"}
Sep 16 12:49:06.932300 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:06.932222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:49:06.932463 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:06.932352    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:49:06.932463 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:06.932413    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls podName:497b9dd9-4306-4db6-92de-958ee5067c9a nodeName:}" failed. No retries permitted until 2026-09-16 12:49:38.932394896 +0000 UTC m=+100.113537538 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls") pod "dns-default-wfs82" (UID: "497b9dd9-4306-4db6-92de-958ee5067c9a") : secret "dns-default-metrics-tls" not found
Sep 16 12:49:10.636151 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:10.636112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l2cs6" event={"ID":"7413c4f8-aa67-4b5f-b144-4a25624ea6ce","Type":"ContainerStarted","Data":"9f065ae7a9ea0ba1305c17c2def7eed358de6987130e5ae683b157650c4859bc"}
Sep 16 12:49:10.636586 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:10.636211    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:49:10.637664 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:10.637621    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ncqr5" event={"ID":"91daa16b-8a95-4961-a53d-81125b48ceaa","Type":"ContainerStarted","Data":"9d22a39a24ca74866e395b2b31cc6f40982f87f7a4e057fcc27d1affc7a4774d"}
Sep 16 12:49:10.656195 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:10.656155    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l2cs6" podStartSLOduration=66.354246145 podStartE2EDuration="1m10.656142358s" totalImagesPullingTime="4.301896213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:10.655808766 +0000 UTC m=+71.836951419" watchObservedRunningTime="2026-09-16 12:49:10.656142358 +0000 UTC m=+71.837284989"
Sep 16 12:49:10.680521 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:10.680486    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ncqr5" podStartSLOduration=66.059761477 podStartE2EDuration="1m10.680473856s" totalImagesPullingTime="4.620712379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:10.67987105 +0000 UTC m=+71.861013702" watchObservedRunningTime="2026-09-16 12:49:10.680473856 +0000 UTC m=+71.861616503"
Sep 16 12:49:11.095487 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.095418    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"]
Sep 16 12:49:11.097087 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.097073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.099865 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.099845    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 12:49:11.099865 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.099856    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 12:49:11.099865 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.099861    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-86cg7\""
Sep 16 12:49:11.099865 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.099856    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:49:11.100159 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.100041    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 12:49:11.108964 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.108945    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-xt4lw"]
Sep 16 12:49:11.110576 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.110556    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.111593 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.111569    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d"]
Sep 16 12:49:11.113207 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.113188    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"]
Sep 16 12:49:11.113302 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.113267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d"
Sep 16 12:49:11.114300 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.114282    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 12:49:11.114408 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.114376    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cggr8\""
Sep 16 12:49:11.114507 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.114492    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:49:11.114722 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.114707    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:49:11.114791 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.114751    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 12:49:11.119237 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.119215    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:49:11.119764 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.119743    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cq6p4\""
Sep 16 12:49:11.120788 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.120775    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 12:49:11.123737 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.123717    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 12:49:11.131648 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.131627    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-xt4lw"]
Sep 16 12:49:11.148692 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.148670    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d"]
Sep 16 12:49:11.163151 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/daf832f1-d96a-415c-8eb3-d6302f20b521-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-sfgdh\" (UID: \"daf832f1-d96a-415c-8eb3-d6302f20b521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.163251 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kqz\" (UniqueName: \"kubernetes.io/projected/6ae74286-8945-465a-9d8b-5f3ed411bade-kube-api-access-62kqz\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.163251 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97wm\" (UniqueName: \"kubernetes.io/projected/daf832f1-d96a-415c-8eb3-d6302f20b521-kube-api-access-m97wm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-sfgdh\" (UID: \"daf832f1-d96a-415c-8eb3-d6302f20b521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.163251 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163202    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ae74286-8945-465a-9d8b-5f3ed411bade-tmp\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.163251 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ae74286-8945-465a-9d8b-5f3ed411bade-service-ca-bundle\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.163464 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6ae74286-8945-465a-9d8b-5f3ed411bade-snapshots\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.163464 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/daf832f1-d96a-415c-8eb3-d6302f20b521-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-sfgdh\" (UID: \"daf832f1-d96a-415c-8eb3-d6302f20b521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.163464 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ae74286-8945-465a-9d8b-5f3ed411bade-serving-cert\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.163464 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqj8f\" (UniqueName: \"kubernetes.io/projected/0ad7cdf9-0336-44ab-ac22-000333386b12-kube-api-access-lqj8f\") pod \"volume-data-source-validator-86695b7cc7-dxs4d\" (UID: \"0ad7cdf9-0336-44ab-ac22-000333386b12\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d"
Sep 16 12:49:11.163464 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.163449    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ae74286-8945-465a-9d8b-5f3ed411bade-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.200354 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.200320    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"]
Sep 16 12:49:11.202171 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.202156    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fc58f76c4-4jhwm"]
Sep 16 12:49:11.202276 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.202259    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:11.203954 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.203937    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.204935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.204920    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 12:49:11.205068 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.205053    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:49:11.205114 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.205078    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rm4dj\""
Sep 16 12:49:11.208536 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.205629    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 12:49:11.209139 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.209117    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:49:11.209240 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.209164    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:49:11.209240 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.209164    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-27ln7\""
Sep 16 12:49:11.209635 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.209617    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:49:11.214329 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.214309    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:49:11.216917 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.216895    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"]
Sep 16 12:49:11.222844 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.222824    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fc58f76c4-4jhwm"]
Sep 16 12:49:11.264094 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ae74286-8945-465a-9d8b-5f3ed411bade-tmp\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.264192 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ae74286-8945-465a-9d8b-5f3ed411bade-service-ca-bundle\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.264192 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-trusted-ca\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.264288 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:11.264288 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264262    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-image-registry-private-configuration\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.264378 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6ae74286-8945-465a-9d8b-5f3ed411bade-snapshots\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.264378 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.264378 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj5z4\" (UniqueName: \"kubernetes.io/projected/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-kube-api-access-hj5z4\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:11.264378 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264349    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-ca-trust-extracted\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.264378 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264372    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-installation-pull-secrets\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.264578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-certificates\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.264578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/daf832f1-d96a-415c-8eb3-d6302f20b521-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-sfgdh\" (UID: \"daf832f1-d96a-415c-8eb3-d6302f20b521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.264578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ae74286-8945-465a-9d8b-5f3ed411bade-serving-cert\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.264578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264453    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ae74286-8945-465a-9d8b-5f3ed411bade-tmp\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.264578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqj8f\" (UniqueName: \"kubernetes.io/projected/0ad7cdf9-0336-44ab-ac22-000333386b12-kube-api-access-lqj8f\") pod \"volume-data-source-validator-86695b7cc7-dxs4d\" (UID: \"0ad7cdf9-0336-44ab-ac22-000333386b12\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d"
Sep 16 12:49:11.264578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ae74286-8945-465a-9d8b-5f3ed411bade-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.264578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264510    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-bound-sa-token\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.264578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/daf832f1-d96a-415c-8eb3-d6302f20b521-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-sfgdh\" (UID: \"daf832f1-d96a-415c-8eb3-d6302f20b521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.264578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kqz\" (UniqueName: \"kubernetes.io/projected/6ae74286-8945-465a-9d8b-5f3ed411bade-kube-api-access-62kqz\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.265006 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97wm\" (UniqueName: \"kubernetes.io/projected/daf832f1-d96a-415c-8eb3-d6302f20b521-kube-api-access-m97wm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-sfgdh\" (UID: \"daf832f1-d96a-415c-8eb3-d6302f20b521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.265006 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2rj\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-kube-api-access-zf2rj\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.265006 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264712    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ae74286-8945-465a-9d8b-5f3ed411bade-service-ca-bundle\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.265006 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.264933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6ae74286-8945-465a-9d8b-5f3ed411bade-snapshots\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.265123 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.265022    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/daf832f1-d96a-415c-8eb3-d6302f20b521-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-sfgdh\" (UID: \"daf832f1-d96a-415c-8eb3-d6302f20b521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.265383 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.265367    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ae74286-8945-465a-9d8b-5f3ed411bade-trusted-ca-bundle\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.267020 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.267004    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/daf832f1-d96a-415c-8eb3-d6302f20b521-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-sfgdh\" (UID: \"daf832f1-d96a-415c-8eb3-d6302f20b521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.267097 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.267076    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ae74286-8945-465a-9d8b-5f3ed411bade-serving-cert\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.273534 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.273507    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97wm\" (UniqueName: \"kubernetes.io/projected/daf832f1-d96a-415c-8eb3-d6302f20b521-kube-api-access-m97wm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-sfgdh\" (UID: \"daf832f1-d96a-415c-8eb3-d6302f20b521\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.273673 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.273635    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqj8f\" (UniqueName: \"kubernetes.io/projected/0ad7cdf9-0336-44ab-ac22-000333386b12-kube-api-access-lqj8f\") pod \"volume-data-source-validator-86695b7cc7-dxs4d\" (UID: \"0ad7cdf9-0336-44ab-ac22-000333386b12\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d"
Sep 16 12:49:11.273962 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.273937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kqz\" (UniqueName: \"kubernetes.io/projected/6ae74286-8945-465a-9d8b-5f3ed411bade-kube-api-access-62kqz\") pod \"insights-operator-5844b856db-xt4lw\" (UID: \"6ae74286-8945-465a-9d8b-5f3ed411bade\") " pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.346418 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.346363    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:11.365852 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.365833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2rj\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-kube-api-access-zf2rj\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.365916 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.365860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-trusted-ca\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.365916 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.365878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:11.365916 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.365895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-image-registry-private-configuration\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.366032 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.365919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.366032 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.365955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj5z4\" (UniqueName: \"kubernetes.io/projected/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-kube-api-access-hj5z4\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:11.366032 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.365980    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-ca-trust-extracted\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.366032 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.365989    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:49:11.366032 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.366011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-installation-pull-secrets\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.366250 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.366046    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls podName:e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:11.866028387 +0000 UTC m=+73.047171022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-m2cwz" (UID: "e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35") : secret "samples-operator-tls" not found
Sep 16 12:49:11.366250 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.366079    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-certificates\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.366250 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.366124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-bound-sa-token\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.366250 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.366082    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:49:11.366250 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.366183    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fc58f76c4-4jhwm: secret "image-registry-tls" not found
Sep 16 12:49:11.366250 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.366232    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls podName:62b1ce98-99da-4bc5-a8d7-d8e08af3111a nodeName:}" failed. No retries permitted until 2026-09-16 12:49:11.866214838 +0000 UTC m=+73.047357490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls") pod "image-registry-6fc58f76c4-4jhwm" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a") : secret "image-registry-tls" not found
Sep 16 12:49:11.366546 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.366448    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-ca-trust-extracted\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.366546 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.366531    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-certificates\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.366907 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.366889    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-trusted-ca\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.368154 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.368137    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-installation-pull-secrets\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.368334 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.368316    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-image-registry-private-configuration\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.375556 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.375530    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj5z4\" (UniqueName: \"kubernetes.io/projected/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-kube-api-access-hj5z4\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:11.375718 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.375702    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-bound-sa-token\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.376677 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.376633    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2rj\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-kube-api-access-zf2rj\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.406187 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.406155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"
Sep 16 12:49:11.420774 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.420751    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-xt4lw"
Sep 16 12:49:11.425171 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.425149    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d"
Sep 16 12:49:11.482078 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.481991    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:11.680178 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.680150    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh"]
Sep 16 12:49:11.683468 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:11.683439    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf832f1_d96a_415c_8eb3_d6302f20b521.slice/crio-391984e3e0e1f2b21e6aa380d42a1eacf51e2217643a791718b86b52cc8c09ef WatchSource:0}: Error finding container 391984e3e0e1f2b21e6aa380d42a1eacf51e2217643a791718b86b52cc8c09ef: Status 404 returned error can't find the container with id 391984e3e0e1f2b21e6aa380d42a1eacf51e2217643a791718b86b52cc8c09ef
Sep 16 12:49:11.705602 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.705579    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-xt4lw"]
Sep 16 12:49:11.708001 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.707978    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d"]
Sep 16 12:49:11.708481 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:11.708457    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae74286_8945_465a_9d8b_5f3ed411bade.slice/crio-878a12d5bf8bee6c06c7dec0d7a9e6d4b5cabba253df06deefda41c4c7188579 WatchSource:0}: Error finding container 878a12d5bf8bee6c06c7dec0d7a9e6d4b5cabba253df06deefda41c4c7188579: Status 404 returned error can't find the container with id 878a12d5bf8bee6c06c7dec0d7a9e6d4b5cabba253df06deefda41c4c7188579
Sep 16 12:49:11.711622 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:11.711602    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad7cdf9_0336_44ab_ac22_000333386b12.slice/crio-cabb18551c43227571bbf3913e72f1d9a290d1d764840d3ac55d6bbc9b871cd3 WatchSource:0}: Error finding container cabb18551c43227571bbf3913e72f1d9a290d1d764840d3ac55d6bbc9b871cd3: Status 404 returned error can't find the container with id cabb18551c43227571bbf3913e72f1d9a290d1d764840d3ac55d6bbc9b871cd3
Sep 16 12:49:11.870003 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.869976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:11.870121 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:11.870011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:11.870121 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.870098    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:49:11.870121 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.870099    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:49:11.870121 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.870109    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fc58f76c4-4jhwm: secret "image-registry-tls" not found
Sep 16 12:49:11.870273 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.870154    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls podName:e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:12.870139301 +0000 UTC m=+74.051281932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-m2cwz" (UID: "e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35") : secret "samples-operator-tls" not found
Sep 16 12:49:11.870273 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:11.870167    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls podName:62b1ce98-99da-4bc5-a8d7-d8e08af3111a nodeName:}" failed. No retries permitted until 2026-09-16 12:49:12.870161433 +0000 UTC m=+74.051304064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls") pod "image-registry-6fc58f76c4-4jhwm" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a") : secret "image-registry-tls" not found
Sep 16 12:49:12.645984 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:12.645898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d" event={"ID":"0ad7cdf9-0336-44ab-ac22-000333386b12","Type":"ContainerStarted","Data":"cabb18551c43227571bbf3913e72f1d9a290d1d764840d3ac55d6bbc9b871cd3"}
Sep 16 12:49:12.647932 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:12.647898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xt4lw" event={"ID":"6ae74286-8945-465a-9d8b-5f3ed411bade","Type":"ContainerStarted","Data":"878a12d5bf8bee6c06c7dec0d7a9e6d4b5cabba253df06deefda41c4c7188579"}
Sep 16 12:49:12.649399 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:12.649376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh" event={"ID":"daf832f1-d96a-415c-8eb3-d6302f20b521","Type":"ContainerStarted","Data":"391984e3e0e1f2b21e6aa380d42a1eacf51e2217643a791718b86b52cc8c09ef"}
Sep 16 12:49:12.879311 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:12.879280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:12.879759 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:12.879325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:12.879759 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:12.879432    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:49:12.879759 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:12.879435    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:49:12.879759 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:12.879517    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fc58f76c4-4jhwm: secret "image-registry-tls" not found
Sep 16 12:49:12.879759 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:12.879505    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls podName:e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.879483647 +0000 UTC m=+76.060626280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-m2cwz" (UID: "e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35") : secret "samples-operator-tls" not found
Sep 16 12:49:12.879759 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:12.879582    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls podName:62b1ce98-99da-4bc5-a8d7-d8e08af3111a nodeName:}" failed. No retries permitted until 2026-09-16 12:49:14.87956321 +0000 UTC m=+76.060705845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls") pod "image-registry-6fc58f76c4-4jhwm" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a") : secret "image-registry-tls" not found
Sep 16 12:49:13.654615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:13.654575    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d" event={"ID":"0ad7cdf9-0336-44ab-ac22-000333386b12","Type":"ContainerStarted","Data":"5406a4775edf292a3dbe177311a5ef513decb6b78ec74414f1217fd404dae931"}
Sep 16 12:49:13.678217 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:13.678164    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dxs4d" podStartSLOduration=1.478107571 podStartE2EDuration="2.678142736s" totalImagesPullingTime="1.200035165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:13.677066425 +0000 UTC m=+74.858209078" watchObservedRunningTime="2026-09-16 12:49:13.678142736 +0000 UTC m=+74.859285389"
Sep 16 12:49:13.837814 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:13.837779    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:14.661892 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:14.661854    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xt4lw" event={"ID":"6ae74286-8945-465a-9d8b-5f3ed411bade","Type":"ContainerStarted","Data":"137d41cd2506d403f00563f663904aebcd4bb8f7e77baa3ff48c7a0ecde76de2"}
Sep 16 12:49:14.663485 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:14.663458    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh" event={"ID":"daf832f1-d96a-415c-8eb3-d6302f20b521","Type":"ContainerStarted","Data":"1f95303699ad02899b3c524cb3ce4ebf904058c2762a05f6cbe91ffeb4890588"}
Sep 16 12:49:14.680109 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:14.680066    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-xt4lw" podStartSLOduration=1.423153488 podStartE2EDuration="3.680052733s" totalImagesPullingTime="2.256899245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:14.679574424 +0000 UTC m=+75.860717078" watchObservedRunningTime="2026-09-16 12:49:14.680052733 +0000 UTC m=+75.861195387"
Sep 16 12:49:14.701296 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:14.701248    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh" podStartSLOduration=1.417734238 podStartE2EDuration="3.701234665s" totalImagesPullingTime="2.283500427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:14.701027701 +0000 UTC m=+75.882170355" watchObservedRunningTime="2026-09-16 12:49:14.701234665 +0000 UTC m=+75.882377317"
Sep 16 12:49:14.898789 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:14.898752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:14.898970 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:14.898796    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:14.898970 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:14.898891    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:49:14.898970 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:14.898901    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fc58f76c4-4jhwm: secret "image-registry-tls" not found
Sep 16 12:49:14.898970 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:14.898902    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:49:14.898970 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:14.898948    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls podName:62b1ce98-99da-4bc5-a8d7-d8e08af3111a nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.898935265 +0000 UTC m=+80.080077897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls") pod "image-registry-6fc58f76c4-4jhwm" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a") : secret "image-registry-tls" not found
Sep 16 12:49:14.898970 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:14.898960    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls podName:e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.898954277 +0000 UTC m=+80.080096907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-m2cwz" (UID: "e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35") : secret "samples-operator-tls" not found
Sep 16 12:49:14.958443 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:14.958365    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:15.082206 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:15.082171    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:15.200456 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:15.200394    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:15.306413 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:15.306294    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:16.982017 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:16.981972    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kfjm9"]
Sep 16 12:49:16.984758 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:16.984736    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:16.988910 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:16.988883    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:49:16.989030 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:16.989011    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bq7jb\""
Sep 16 12:49:16.991211 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:16.991194    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:49:17.002199 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.002174    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kfjm9"]
Sep 16 12:49:17.096380 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.096356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:17.116384 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.116362    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63d8096b-7afb-4492-bf16-06c4c5e19de2-crio-socket\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.116435 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.116401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9nxv\" (UniqueName: \"kubernetes.io/projected/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-api-access-c9nxv\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.116435 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.116428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63d8096b-7afb-4492-bf16-06c4c5e19de2-data-volume\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.116521 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.116497    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.116563 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.116522    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.217362 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.217337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63d8096b-7afb-4492-bf16-06c4c5e19de2-crio-socket\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.217458 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.217373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9nxv\" (UniqueName: \"kubernetes.io/projected/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-api-access-c9nxv\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.217458 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.217410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63d8096b-7afb-4492-bf16-06c4c5e19de2-data-volume\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.217458 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.217442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.217579 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.217460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63d8096b-7afb-4492-bf16-06c4c5e19de2-crio-socket\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.217579 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.217467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.217579 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:17.217549    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:17.217702 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:17.217580    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:17.217702 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:17.217609    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:17.717592233 +0000 UTC m=+78.898734868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:17.217702 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:17.217675    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:17.717636751 +0000 UTC m=+78.898779404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:17.217845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.217828    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63d8096b-7afb-4492-bf16-06c4c5e19de2-data-volume\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.239164 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.239114    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9nxv\" (UniqueName: \"kubernetes.io/projected/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-api-access-c9nxv\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.275394 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.275374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:17.412627 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.412601    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:17.514199 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.514113    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:17.722278 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.722241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.722278 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:17.722282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:17.722567 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:17.722393    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:17.722567 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:17.722429    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:17.722567 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:17.722453    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.722438885 +0000 UTC m=+79.903581515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:17.722567 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:17.722476    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:18.722463397 +0000 UTC m=+79.903606028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:18.730434 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:18.730404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:18.730894 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:18.730440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:18.730894 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:18.730553    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:18.730894 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:18.730567    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:18.730894 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:18.730603    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:20.730590815 +0000 UTC m=+81.911733446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:18.730894 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:18.730620    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:20.730608209 +0000 UTC m=+81.911750839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:18.931912 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:18.931885    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:18.932038 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:18.931944    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:18.932038 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:18.932020    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:49:18.932117 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:18.932044    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:49:18.932117 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:18.932054    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fc58f76c4-4jhwm: secret "image-registry-tls" not found
Sep 16 12:49:18.932117 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:18.932076    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls podName:e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:26.932060148 +0000 UTC m=+88.113202778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-m2cwz" (UID: "e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35") : secret "samples-operator-tls" not found
Sep 16 12:49:18.932117 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:18.932092    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls podName:62b1ce98-99da-4bc5-a8d7-d8e08af3111a nodeName:}" failed. No retries permitted until 2026-09-16 12:49:26.932081771 +0000 UTC m=+88.113224402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls") pod "image-registry-6fc58f76c4-4jhwm" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a") : secret "image-registry-tls" not found
Sep 16 12:49:19.144251 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:19.144225    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:19.238865 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:19.238843    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:19.488178 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:19.488120    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4p7dg_30264cd8-3720-4fbe-8ed7-cf33859e997b/dns-node-resolver/0.log"
Sep 16 12:49:20.144460 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:20.144430    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:20.687509 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:20.687483    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dw7rd_a61a032b-e590-47e4-b7cd-e73d1ffcf9bb/node-ca/0.log"
Sep 16 12:49:20.742955 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:20.742930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:20.743090 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:20.742961    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:20.743090 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:20.743021    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:20.743090 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:20.743069    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:24.743056216 +0000 UTC m=+85.924198846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:20.743090 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:20.743084    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:20.743264 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:20.743153    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:24.743135477 +0000 UTC m=+85.924278124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:21.180686 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:21.180633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:21.488965 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:21.488892    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kr5tj_0db4d710-77d0-4884-a702-739868c1c6ad/serve-healthcheck-canary/0.log"
Sep 16 12:49:21.534854 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:21.534826    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:21.699283 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:21.699252    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:21.838379 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:21.838319    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:22.179751 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:22.179713    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:22.278914 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:22.278886    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:22.291260 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:22.291240    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-sfgdh_daf832f1-d96a-415c-8eb3-d6302f20b521/kube-storage-version-migrator-operator/0.log"
Sep 16 12:49:22.371517 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:22.371491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:23.313969 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:23.313937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:23.511357 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:23.511332    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:23.905162 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:23.905133    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:24.118638 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:24.118609    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:24.368897 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:24.368871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:24.717684 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:24.717607    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:24.774844 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:24.774821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:24.774898 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:24.774849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:24.774937 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:24.774931    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:49:24.774968 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:24.774955    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:24.775001 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:24.774975    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:32.774963675 +0000 UTC m=+93.956106306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : configmap "kube-rbac-proxy" not found
Sep 16 12:49:24.775001 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:49:24.774999    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls podName:63d8096b-7afb-4492-bf16-06c4c5e19de2 nodeName:}" failed. No retries permitted until 2026-09-16 12:49:32.774986389 +0000 UTC m=+93.956129019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kfjm9" (UID: "63d8096b-7afb-4492-bf16-06c4c5e19de2") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:49:24.874382 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:24.874361    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:24.973474 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:24.973418    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:25.419356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:25.419301    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:25.519430 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:25.519407    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:25.940958 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:25.940930    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:26.102843 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:26.102812    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:26.315430 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:26.315352    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:26.398524 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:26.398499    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:26.744441 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:26.744413    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:26.973490 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:26.973463    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:26.994088 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:26.994064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:26.994239 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:26.994121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:26.996613 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:26.996560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") pod \"image-registry-6fc58f76c4-4jhwm\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") " pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:26.996734 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:26.996683    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-m2cwz\" (UID: \"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:27.115621 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.115578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"
Sep 16 12:49:27.120541 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.120518    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:27.324744 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.324665    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:27.410989 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.410962    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz"]
Sep 16 12:49:27.458804 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.458773    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fc58f76c4-4jhwm"]
Sep 16 12:49:27.461452 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:27.461429    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b1ce98_99da_4bc5_a8d7_d8e08af3111a.slice/crio-471c7181c02c49353aae0b8f1ee7369d4b1a1ef1b2d18f30528a496d55c7f7fe WatchSource:0}: Error finding container 471c7181c02c49353aae0b8f1ee7369d4b1a1ef1b2d18f30528a496d55c7f7fe: Status 404 returned error can't find the container with id 471c7181c02c49353aae0b8f1ee7369d4b1a1ef1b2d18f30528a496d55c7f7fe
Sep 16 12:49:27.703321 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.703290    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm" event={"ID":"62b1ce98-99da-4bc5-a8d7-d8e08af3111a","Type":"ContainerStarted","Data":"12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242"}
Sep 16 12:49:27.703512 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.703336    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm" event={"ID":"62b1ce98-99da-4bc5-a8d7-d8e08af3111a","Type":"ContainerStarted","Data":"471c7181c02c49353aae0b8f1ee7369d4b1a1ef1b2d18f30528a496d55c7f7fe"}
Sep 16 12:49:27.703512 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.703420    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:27.704736 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.704711    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz" event={"ID":"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35","Type":"ContainerStarted","Data":"78115cfd6dbc12f96ba0c4194148a246e1f359c08695c28d0ec68e2b1a20da96"}
Sep 16 12:49:27.731602 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:27.731550    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm" podStartSLOduration=16.731529116 podStartE2EDuration="16.731529116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:27.730310528 +0000 UTC m=+88.911453183" watchObservedRunningTime="2026-09-16 12:49:27.731529116 +0000 UTC m=+88.912671771"
Sep 16 12:49:28.108685 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:28.108564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:28.291006 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:28.290969    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:28.495821 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:28.495794    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:28.606439 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:28.606404    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:28.713907 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:28.713866    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:28.908094 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:28.908066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:29.098664 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:29.098619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:29.712660 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:29.712616    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz" event={"ID":"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35","Type":"ContainerStarted","Data":"74b2d16ff81b38b1f36deb8efebad20c4ffad74bf4d7ef3a5ad1046dce3b41a9"}
Sep 16 12:49:29.712995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:29.712678    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz" event={"ID":"e0bdd9bb-41fe-4bb6-88e4-84926c2a2f35","Type":"ContainerStarted","Data":"317ec94e86615d29066829f7971d9023f510401f4e979228adf05c70101b1fa3"}
Sep 16 12:49:29.736763 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:29.736726    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-m2cwz" podStartSLOduration=17.243142501 podStartE2EDuration="18.736711475s" totalImagesPullingTime="1.493568974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:29.734950089 +0000 UTC m=+90.916092742" watchObservedRunningTime="2026-09-16 12:49:29.736711475 +0000 UTC m=+90.917854119"
Sep 16 12:49:30.106884 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:30.106817    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:30.502260 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:30.502233    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:30.693854 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:30.693814    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:30.900663 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:30.900620    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:31.302084 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:31.301964    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:31.701881 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:31.701824    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:31.930506 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:31.930457    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:32.841703 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:32.841670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:32.841703 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:32.841706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:32.842165 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:32.842142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63d8096b-7afb-4492-bf16-06c4c5e19de2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:32.844063 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:32.844043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63d8096b-7afb-4492-bf16-06c4c5e19de2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfjm9\" (UID: \"63d8096b-7afb-4492-bf16-06c4c5e19de2\") " pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:32.894069 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:32.894048    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kfjm9"
Sep 16 12:49:33.170043 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:33.169984    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kfjm9"]
Sep 16 12:49:33.172488 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:33.172470    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d8096b_7afb_4492_bf16_06c4c5e19de2.slice/crio-9e12f141a5333fd258f61b62a341760d716c7535fc6bb3179a398a454adf40fa WatchSource:0}: Error finding container 9e12f141a5333fd258f61b62a341760d716c7535fc6bb3179a398a454adf40fa: Status 404 returned error can't find the container with id 9e12f141a5333fd258f61b62a341760d716c7535fc6bb3179a398a454adf40fa
Sep 16 12:49:33.724875 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:33.724840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfjm9" event={"ID":"63d8096b-7afb-4492-bf16-06c4c5e19de2","Type":"ContainerStarted","Data":"8542cc89e41bd4086939f2ea7dbaa8de96cb12886a93bd1fdad32dfde31bb6b6"}
Sep 16 12:49:33.725033 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:33.724900    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfjm9" event={"ID":"63d8096b-7afb-4492-bf16-06c4c5e19de2","Type":"ContainerStarted","Data":"9e12f141a5333fd258f61b62a341760d716c7535fc6bb3179a398a454adf40fa"}
Sep 16 12:49:33.944282 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:33.944248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:34.045165 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:34.045142    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:34.730130 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:34.730099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfjm9" event={"ID":"63d8096b-7afb-4492-bf16-06c4c5e19de2","Type":"ContainerStarted","Data":"7697ea89a1e4fedd843e5a2d98956619cb4164d99ada7234cddfdc6d9e277ed3"}
Sep 16 12:49:35.113812 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.113772    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:35.220794 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.220767    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:35.322790 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.322766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:35.421190 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.421127    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:35.584026 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.583998    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:35.685129 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.685068    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:35.737880 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.737838    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfjm9" event={"ID":"63d8096b-7afb-4492-bf16-06c4c5e19de2","Type":"ContainerStarted","Data":"ca10cb561e22fa2cedbde9bc541567bf7c9f4966821512adcb7fa9a42acad94d"}
Sep 16 12:49:35.761758 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.761720    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kfjm9" podStartSLOduration=17.784178867 podStartE2EDuration="19.761707112s" totalImagesPullingTime="1.977528245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:35.761625881 +0000 UTC m=+96.942768534" watchObservedRunningTime="2026-09-16 12:49:35.761707112 +0000 UTC m=+96.942849765"
Sep 16 12:49:35.788980 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.788958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:35.867547 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:35.867522    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:36.435193 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:36.435167    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:36.538968 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:36.538946    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:36.648824 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:36.648800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:36.752123 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:36.752067    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:36.878995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:36.878974    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:37.020867 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:37.020795    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:38.848627 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:38.848594    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:38.942300 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:38.942275    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:38.984352 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:38.984331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:49:38.986688 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:38.986664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/497b9dd9-4306-4db6-92de-958ee5067c9a-metrics-tls\") pod \"dns-default-wfs82\" (UID: \"497b9dd9-4306-4db6-92de-958ee5067c9a\") " pod="openshift-dns/dns-default-wfs82"
Sep 16 12:49:39.042616 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:39.042596    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rwp6v\""
Sep 16 12:49:39.045455 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:39.045442    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wfs82"
Sep 16 12:49:39.321361 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:39.321312    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wfs82"]
Sep 16 12:49:39.323943 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:39.323901    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497b9dd9_4306_4db6_92de_958ee5067c9a.slice/crio-880cbb2d9a24e2f18c69f9fce773e860b366eeb4757780182c50905d99f61934 WatchSource:0}: Error finding container 880cbb2d9a24e2f18c69f9fce773e860b366eeb4757780182c50905d99f61934: Status 404 returned error can't find the container with id 880cbb2d9a24e2f18c69f9fce773e860b366eeb4757780182c50905d99f61934
Sep 16 12:49:39.752634 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:39.752601    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wfs82" event={"ID":"497b9dd9-4306-4db6-92de-958ee5067c9a","Type":"ContainerStarted","Data":"880cbb2d9a24e2f18c69f9fce773e860b366eeb4757780182c50905d99f61934"}
Sep 16 12:49:41.642506 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:41.642482    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l2cs6"
Sep 16 12:49:41.760558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:41.760498    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wfs82" event={"ID":"497b9dd9-4306-4db6-92de-958ee5067c9a","Type":"ContainerStarted","Data":"6b13da57ada55364760b9092d2ea2cf72a7a81615fcdaabc23f81b8737ba157f"}
Sep 16 12:49:41.760558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:41.760540    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wfs82" event={"ID":"497b9dd9-4306-4db6-92de-958ee5067c9a","Type":"ContainerStarted","Data":"b605167b27eedac8525a624abd044e05ceda12adfd0fada2a9e5ebe225047c38"}
Sep 16 12:49:41.760736 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:41.760667    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wfs82"
Sep 16 12:49:41.790488 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:41.790421    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wfs82" podStartSLOduration=64.889115835 podStartE2EDuration="1m6.790407918s" totalImagesPullingTime="1.901292083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:41.788995618 +0000 UTC m=+102.970138281" watchObservedRunningTime="2026-09-16 12:49:41.790407918 +0000 UTC m=+102.971550571"
Sep 16 12:49:42.128628 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.128605    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"]
Sep 16 12:49:42.149608 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.149588    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.149772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.149595    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"]
Sep 16 12:49:42.153755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.153736    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 12:49:42.154001 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.153985    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 12:49:42.154728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.154667    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 12:49:42.154796 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.154692    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 12:49:42.154891 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.154872    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 12:49:42.156219 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.156112    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 12:49:42.156219 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.156144    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 12:49:42.157724 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.157706    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fc58f76c4-4jhwm"]
Sep 16 12:49:42.162066 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.162041    2575 patch_prober.go:28] interesting pod/image-registry-6fc58f76c4-4jhwm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 12:49:42.162153 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.162081    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm" podUID="62b1ce98-99da-4bc5-a8d7-d8e08af3111a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:49:42.274858 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.274828    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:42.312114 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.312087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.312227 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.312125    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-hub\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.312227 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.312152    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtfb\" (UniqueName: \"kubernetes.io/projected/5831727e-6e94-4eec-a2b6-e35ecf5ef542-kube-api-access-hhtfb\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.312339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.312227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5831727e-6e94-4eec-a2b6-e35ecf5ef542-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.312339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.312279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-ca\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.312339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.312324    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.365969 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.365935    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:42.413631 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.413567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.413631 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.413607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.413831 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.413636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-hub\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.413831 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.413690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtfb\" (UniqueName: \"kubernetes.io/projected/5831727e-6e94-4eec-a2b6-e35ecf5ef542-kube-api-access-hhtfb\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.413831 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.413709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5831727e-6e94-4eec-a2b6-e35ecf5ef542-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.413831 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.413730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-ca\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.414679 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.414632    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5831727e-6e94-4eec-a2b6-e35ecf5ef542-additional-service-ca\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.416381 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.416358    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.416479 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.416423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-ca\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.416479 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.416469    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.416557 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.416486    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5831727e-6e94-4eec-a2b6-e35ecf5ef542-hub\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.422558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.422535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtfb\" (UniqueName: \"kubernetes.io/projected/5831727e-6e94-4eec-a2b6-e35ecf5ef542-kube-api-access-hhtfb\") pod \"cluster-proxy-proxy-agent-67868fc9bf-wrfv7\" (UID: \"5831727e-6e94-4eec-a2b6-e35ecf5ef542\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.476069 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.476047    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"
Sep 16 12:49:42.766468 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:42.766397    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7"]
Sep 16 12:49:42.769262 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:42.769240    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5831727e_6e94_4eec_a2b6_e35ecf5ef542.slice/crio-cbf376349d51f189d74f7d41f50075295c89a3416993c96330b015cba0069bdd WatchSource:0}: Error finding container cbf376349d51f189d74f7d41f50075295c89a3416993c96330b015cba0069bdd: Status 404 returned error can't find the container with id cbf376349d51f189d74f7d41f50075295c89a3416993c96330b015cba0069bdd
Sep 16 12:49:43.767826 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:43.767787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7" event={"ID":"5831727e-6e94-4eec-a2b6-e35ecf5ef542","Type":"ContainerStarted","Data":"cbf376349d51f189d74f7d41f50075295c89a3416993c96330b015cba0069bdd"}
Sep 16 12:49:45.473962 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:45.473924    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:45.569711 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:45.569616    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:46.875830 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:46.875801    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:46.978475 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:46.978449    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:47.082963 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:47.082914    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:47.178040 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:47.177958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:47.783470 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:47.783435    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7" event={"ID":"5831727e-6e94-4eec-a2b6-e35ecf5ef542","Type":"ContainerStarted","Data":"b2c253b0a881d3bc30881bbc9fcfd3e1c00d204ec14c4ab17a7284f0f689c090"}
Sep 16 12:49:47.783676 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:47.783475    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7" event={"ID":"5831727e-6e94-4eec-a2b6-e35ecf5ef542","Type":"ContainerStarted","Data":"46ea1103cbc18197f39926d671a074217c01500b54b777b83b03e8b56e84a5cb"}
Sep 16 12:49:47.783676 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:47.783492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7" event={"ID":"5831727e-6e94-4eec-a2b6-e35ecf5ef542","Type":"ContainerStarted","Data":"55766040f6e1c0127be601d8d37cbabd60533214548b294f50206f24742ee67a"}
Sep 16 12:49:47.807203 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:47.807160    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-67868fc9bf-wrfv7" podStartSLOduration=1.6699789379999999 podStartE2EDuration="5.807148412s" totalImagesPullingTime="4.137169474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:47.807013853 +0000 UTC m=+108.988156505" watchObservedRunningTime="2026-09-16 12:49:47.807148412 +0000 UTC m=+108.988291065"
Sep 16 12:49:49.609585 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:49.609552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:49.711271 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:49.711240    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:50.133939 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:50.133910    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:50.235615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:50.235587    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:50.347793 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:50.347762    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:51.766078 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:51.766054    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wfs82"
Sep 16 12:49:52.162687 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:52.162662    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:49:52.253935 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:52.253908    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:52.353545 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:52.353516    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:52.419521 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:52.419457    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:53.413818 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:53.413787    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:53.492442 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:53.492413    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:53.575203 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:53.575163    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:54.253962 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.253928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:54.537531 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.537427    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:54.568968 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.568947    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mh66z"]
Sep 16 12:49:54.571123 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.571108    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.574795 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.574778    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:49:54.574958 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.574941    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:49:54.575194 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.575178    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:49:54.580018 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.580001    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:49:54.580810 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.580791    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ksdcc\""
Sep 16 12:49:54.580880 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.580836    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:49:54.580880 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.580845    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:49:54.639358 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.639339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:54.702565 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.702538    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-tls\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.702679 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.702577    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z2c7\" (UniqueName: \"kubernetes.io/projected/937faa2a-954c-4136-8ebe-175b092bffe3-kube-api-access-7z2c7\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.702679 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.702621    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/937faa2a-954c-4136-8ebe-175b092bffe3-metrics-client-ca\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.702679 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.702666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/937faa2a-954c-4136-8ebe-175b092bffe3-root\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.702852 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.702706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.702852 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.702736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/937faa2a-954c-4136-8ebe-175b092bffe3-sys\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.702852 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.702775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-textfile\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.702852 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.702829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-wtmp\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.703018 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.702874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.738131 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.738108    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:54.803546 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803484    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/937faa2a-954c-4136-8ebe-175b092bffe3-sys\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.803546 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-textfile\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.803546 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803538    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-wtmp\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.803772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.803772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803586    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/937faa2a-954c-4136-8ebe-175b092bffe3-sys\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.803772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-tls\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.803772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803634    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2c7\" (UniqueName: \"kubernetes.io/projected/937faa2a-954c-4136-8ebe-175b092bffe3-kube-api-access-7z2c7\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.803772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803699    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/937faa2a-954c-4136-8ebe-175b092bffe3-metrics-client-ca\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.803772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803705    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-wtmp\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.804134 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/937faa2a-954c-4136-8ebe-175b092bffe3-root\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.804134 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.804134 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803866    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-textfile\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.804134 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.803919    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/937faa2a-954c-4136-8ebe-175b092bffe3-root\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.804367 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.804342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.804441 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.804422    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/937faa2a-954c-4136-8ebe-175b092bffe3-metrics-client-ca\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.806126 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.806107    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-tls\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.806319 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.806290    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/937faa2a-954c-4136-8ebe-175b092bffe3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.812193 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.812177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z2c7\" (UniqueName: \"kubernetes.io/projected/937faa2a-954c-4136-8ebe-175b092bffe3-kube-api-access-7z2c7\") pod \"node-exporter-mh66z\" (UID: \"937faa2a-954c-4136-8ebe-175b092bffe3\") " pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.838897 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.838878    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:54.879071 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.879047    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mh66z"
Sep 16 12:49:54.887702 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:54.887680    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937faa2a_954c_4136_8ebe_175b092bffe3.slice/crio-1fa5274edc342a8f543c5bb99be1fb77ce70d02b54c7cfca62ea9192c1aa97b5 WatchSource:0}: Error finding container 1fa5274edc342a8f543c5bb99be1fb77ce70d02b54c7cfca62ea9192c1aa97b5: Status 404 returned error can't find the container with id 1fa5274edc342a8f543c5bb99be1fb77ce70d02b54c7cfca62ea9192c1aa97b5
Sep 16 12:49:54.938730 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:54.938706    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:55.053755 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:55.053686    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:55.191119 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:55.191093    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:55.502845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:55.502804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:55.676765 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:55.676729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:55.813710 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:55.813677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh66z" event={"ID":"937faa2a-954c-4136-8ebe-175b092bffe3","Type":"ContainerStarted","Data":"1fa5274edc342a8f543c5bb99be1fb77ce70d02b54c7cfca62ea9192c1aa97b5"}
Sep 16 12:49:56.047418 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:56.047345    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:56.116224 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:56.116197    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:56.373367 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:56.373337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:56.818570 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:56.818486    2575 generic.go:356] "Generic (PLEG): container finished" podID="937faa2a-954c-4136-8ebe-175b092bffe3" containerID="0600f2057546a4f2a5a308cf114aa9a1cfc4d8d161277948eb41fda77203007a" exitCode=0
Sep 16 12:49:56.818570 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:56.818539    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh66z" event={"ID":"937faa2a-954c-4136-8ebe-175b092bffe3","Type":"ContainerDied","Data":"0600f2057546a4f2a5a308cf114aa9a1cfc4d8d161277948eb41fda77203007a"}
Sep 16 12:49:57.515819 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.515784    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:57.618346 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.618314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:57.642732 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.642705    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7cf89f7648-g84th"]
Sep 16 12:49:57.645097 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.645079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.647906 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.647882    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 12:49:57.647906 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.647905    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-kj247\""
Sep 16 12:49:57.648150 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.648135    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 12:49:57.648224 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.648165    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 12:49:57.648282 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.648252    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-95dqk2ntkln3e\""
Sep 16 12:49:57.648412 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.648397    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 12:49:57.649062 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.649049    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 12:49:57.663516 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.663491    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7cf89f7648-g84th"]
Sep 16 12:49:57.723042 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.723017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-metrics-client-ca\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.723211 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.723067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.723211 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.723093    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-tls\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.723211 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.723197    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.723387 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.723227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-grpc-tls\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.723387 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.723274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7q25\" (UniqueName: \"kubernetes.io/projected/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-kube-api-access-m7q25\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.723387 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.723306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.723387 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.723378    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.823789 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.823707    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.823789 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.823756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-metrics-client-ca\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.824272 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.823792    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.824272 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.823818    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-tls\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.824272 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.823855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.824272 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.823977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-grpc-tls\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.824272 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.824037    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7q25\" (UniqueName: \"kubernetes.io/projected/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-kube-api-access-m7q25\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.824272 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.824071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.824603 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.824575    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-metrics-client-ca\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.825023 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.824992    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh66z" event={"ID":"937faa2a-954c-4136-8ebe-175b092bffe3","Type":"ContainerStarted","Data":"6cea6b99295343c473ebebffd423bb48aa135ff8d35b1a1570e2825cb4f4b919"}
Sep 16 12:49:57.825286 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.825055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh66z" event={"ID":"937faa2a-954c-4136-8ebe-175b092bffe3","Type":"ContainerStarted","Data":"eee996d2da97471efd7e6559470fc3eef361e3ec401fe75643c4244a76808a52"}
Sep 16 12:49:57.826772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.826747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.827228 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.827207    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-tls\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.827316 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.827275    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.827316 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.827301    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.827502 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.827482    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.827557 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.827543    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-secret-grpc-tls\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.840519 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.840499    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7q25\" (UniqueName: \"kubernetes.io/projected/27092ddf-a8c7-4ee2-bd2b-656cdee79acb-kube-api-access-m7q25\") pod \"thanos-querier-7cf89f7648-g84th\" (UID: \"27092ddf-a8c7-4ee2-bd2b-656cdee79acb\") " pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:57.854799 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.854763    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mh66z" podStartSLOduration=2.892105772 podStartE2EDuration="3.854750609s" totalImagesPullingTime="889.021965ms" totalInitContainerRuntime="73.622872ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:57.853301749 +0000 UTC m=+119.034444402" watchObservedRunningTime="2026-09-16 12:49:57.854750609 +0000 UTC m=+119.035893262"
Sep 16 12:49:57.954532 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:57.954507    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:49:58.186471 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:58.186443    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:58.229556 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:58.229531    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7cf89f7648-g84th"]
Sep 16 12:49:58.232961 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:49:58.232938    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27092ddf_a8c7_4ee2_bd2b_656cdee79acb.slice/crio-3628b7f0fff9ef8bf355087c9fc52307d3ee89ada392c7ae2ab1cbaaed0bd907 WatchSource:0}: Error finding container 3628b7f0fff9ef8bf355087c9fc52307d3ee89ada392c7ae2ab1cbaaed0bd907: Status 404 returned error can't find the container with id 3628b7f0fff9ef8bf355087c9fc52307d3ee89ada392c7ae2ab1cbaaed0bd907
Sep 16 12:49:58.345079 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:58.345054    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:58.830219 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:58.830183    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th" event={"ID":"27092ddf-a8c7-4ee2-bd2b-656cdee79acb","Type":"ContainerStarted","Data":"3628b7f0fff9ef8bf355087c9fc52307d3ee89ada392c7ae2ab1cbaaed0bd907"}
Sep 16 12:49:59.288786 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:59.288756    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:59.392900 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:59.392849    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:59.518174 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:59.518143    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:59.618224 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:59.618193    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:49:59.725617 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:49:59.725575    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:00.811570 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:00.811543    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:00.839759 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:00.839735    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th" event={"ID":"27092ddf-a8c7-4ee2-bd2b-656cdee79acb","Type":"ContainerStarted","Data":"aad070cbe83ae06cb3b676de17f3f93ea7e58259641f5fc3719f0466b000d8c3"}
Sep 16 12:50:00.839861 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:00.839799    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th" event={"ID":"27092ddf-a8c7-4ee2-bd2b-656cdee79acb","Type":"ContainerStarted","Data":"ff07a50e662f8dfb21f5827623c46cf34511654cda5104b7dd6982dab5437135"}
Sep 16 12:50:00.839861 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:00.839826    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th" event={"ID":"27092ddf-a8c7-4ee2-bd2b-656cdee79acb","Type":"ContainerStarted","Data":"c6f6d1dbd193140b9f692fac92a6b37766d7b11f78e2a0b998b8dac0d6250d01"}
Sep 16 12:50:00.839861 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:00.839854    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th" event={"ID":"27092ddf-a8c7-4ee2-bd2b-656cdee79acb","Type":"ContainerStarted","Data":"92fed5955ad8d14d44000bd914f29caec6dd5281abcf9c2542cbd224381b5d61"}
Sep 16 12:50:01.412263 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.412226    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:01.481231 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.481203    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:01.659666 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.659618    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-786db67b48-dr6sg"]
Sep 16 12:50:01.662507 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.662468    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.665342 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.665321    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:50:01.665476 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.665357    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:50:01.665476 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.665367    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:50:01.665607 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.665485    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c7h5m\""
Sep 16 12:50:01.665607 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.665520    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:50:01.665831 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.665815    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:50:01.665926 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.665912    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:50:01.665971 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.665919    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:50:01.675826 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.675804    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:50:01.676887 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.676869    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786db67b48-dr6sg"]
Sep 16 12:50:01.755023 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.754986    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-oauth-config\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.755191 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.755042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vm9x\" (UniqueName: \"kubernetes.io/projected/01431672-4090-4d29-9116-3bf749f80aee-kube-api-access-2vm9x\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.755191 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.755068    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-serving-cert\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.755191 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.755082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-oauth-serving-cert\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.755191 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.755118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-console-config\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.755191 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.755134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-service-ca\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.755191 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.755148    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-trusted-ca-bundle\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.846363 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.846328    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th" event={"ID":"27092ddf-a8c7-4ee2-bd2b-656cdee79acb","Type":"ContainerStarted","Data":"b04e6bd99bb27447fa9b08a227fa15c1854651d48ab5eb2c7a9ee652022daabe"}
Sep 16 12:50:01.846760 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.846390    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th" event={"ID":"27092ddf-a8c7-4ee2-bd2b-656cdee79acb","Type":"ContainerStarted","Data":"6274a150183cfa70d066cd7543ce3e8bab1f512dbc207dd1c9868e5cbfb8f14a"}
Sep 16 12:50:01.846760 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.846497    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:50:01.856378 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.856358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-console-config\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.856455 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.856386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-service-ca\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.856455 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.856403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-trusted-ca-bundle\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.856639 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.856608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-oauth-config\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.856784 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.856766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vm9x\" (UniqueName: \"kubernetes.io/projected/01431672-4090-4d29-9116-3bf749f80aee-kube-api-access-2vm9x\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.856852 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.856833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-serving-cert\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.856907 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.856855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-oauth-serving-cert\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.857213 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.857188    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-service-ca\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.857385 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.857358    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-console-config\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.857528 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.857507    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-oauth-serving-cert\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.857814 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.857797    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-trusted-ca-bundle\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.859515 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.859484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-oauth-config\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.859733 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.859716    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-serving-cert\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.865523 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.865505    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vm9x\" (UniqueName: \"kubernetes.io/projected/01431672-4090-4d29-9116-3bf749f80aee-kube-api-access-2vm9x\") pod \"console-786db67b48-dr6sg\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") " pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:01.871521 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.871479    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th" podStartSLOduration=2.517314955 podStartE2EDuration="4.87146535s" totalImagesPullingTime="2.354150395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:01.869849344 +0000 UTC m=+123.050991997" watchObservedRunningTime="2026-09-16 12:50:01.87146535 +0000 UTC m=+123.052608002"
Sep 16 12:50:01.976752 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:01.976676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:02.252772 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:02.252678    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786db67b48-dr6sg"]
Sep 16 12:50:02.255462 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:50:02.255441    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01431672_4090_4d29_9116_3bf749f80aee.slice/crio-3db155fe041998338d78f99d94bc3058934876e87335905c7078f26d99d96cf4 WatchSource:0}: Error finding container 3db155fe041998338d78f99d94bc3058934876e87335905c7078f26d99d96cf4: Status 404 returned error can't find the container with id 3db155fe041998338d78f99d94bc3058934876e87335905c7078f26d99d96cf4
Sep 16 12:50:02.497509 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:02.497476    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:02.568874 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:02.568798    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:02.850625 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:02.850536    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786db67b48-dr6sg" event={"ID":"01431672-4090-4d29-9116-3bf749f80aee","Type":"ContainerStarted","Data":"3db155fe041998338d78f99d94bc3058934876e87335905c7078f26d99d96cf4"}
Sep 16 12:50:03.216889 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:03.216855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:03.319880 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:03.319847    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:03.907089 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:03.907057    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:04.002693 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:04.002658    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:04.143419 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:04.143384    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:04.247834 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:04.247754    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:05.863747 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:05.863712    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786db67b48-dr6sg" event={"ID":"01431672-4090-4d29-9116-3bf749f80aee","Type":"ContainerStarted","Data":"d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121"}
Sep 16 12:50:05.882486 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:05.882445    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-786db67b48-dr6sg" podStartSLOduration=1.9869120740000001 podStartE2EDuration="4.882431088s" totalImagesPullingTime="2.895519014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:05.882192825 +0000 UTC m=+127.063335479" watchObservedRunningTime="2026-09-16 12:50:05.882431088 +0000 UTC m=+127.063573740"
Sep 16 12:50:07.176811 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.176756    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm" podUID="62b1ce98-99da-4bc5-a8d7-d8e08af3111a" containerName="registry" containerID="cri-o://12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242" gracePeriod=30
Sep 16 12:50:07.414283 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.414264    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:50:07.506485 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.506414    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-image-registry-private-configuration\") pod \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") "
Sep 16 12:50:07.506485 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.506465    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-ca-trust-extracted\") pod \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") "
Sep 16 12:50:07.506713 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.506511    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-installation-pull-secrets\") pod \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") "
Sep 16 12:50:07.506713 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.506533    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-bound-sa-token\") pod \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") "
Sep 16 12:50:07.506713 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.506570    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf2rj\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-kube-api-access-zf2rj\") pod \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") "
Sep 16 12:50:07.506713 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.506603    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-certificates\") pod \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") "
Sep 16 12:50:07.506713 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.506633    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") pod \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") "
Sep 16 12:50:07.506954 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.506735    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-trusted-ca\") pod \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\" (UID: \"62b1ce98-99da-4bc5-a8d7-d8e08af3111a\") "
Sep 16 12:50:07.507436 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.507076    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "62b1ce98-99da-4bc5-a8d7-d8e08af3111a" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:07.507436 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.507371    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "62b1ce98-99da-4bc5-a8d7-d8e08af3111a" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:50:07.509120 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.509075    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "62b1ce98-99da-4bc5-a8d7-d8e08af3111a" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:07.509250 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.509155    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "62b1ce98-99da-4bc5-a8d7-d8e08af3111a" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:07.509301 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.509262    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "62b1ce98-99da-4bc5-a8d7-d8e08af3111a" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:50:07.509397 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.509381    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "62b1ce98-99da-4bc5-a8d7-d8e08af3111a" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:07.509607 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.509592    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-kube-api-access-zf2rj" (OuterVolumeSpecName: "kube-api-access-zf2rj") pod "62b1ce98-99da-4bc5-a8d7-d8e08af3111a" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a"). InnerVolumeSpecName "kube-api-access-zf2rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:07.515382 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.515360    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "62b1ce98-99da-4bc5-a8d7-d8e08af3111a" (UID: "62b1ce98-99da-4bc5-a8d7-d8e08af3111a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:50:07.607702 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.607672    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-image-registry-private-configuration\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:50:07.607702 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.607696    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-ca-trust-extracted\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:50:07.607702 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.607706    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-installation-pull-secrets\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:50:07.607897 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.607716    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-bound-sa-token\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:50:07.607897 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.607725    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf2rj\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-kube-api-access-zf2rj\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:50:07.607897 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.607734    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-certificates\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:50:07.607897 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.607742    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-registry-tls\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:50:07.607897 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.607749    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62b1ce98-99da-4bc5-a8d7-d8e08af3111a-trusted-ca\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:50:07.856674 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.856593    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7cf89f7648-g84th"
Sep 16 12:50:07.872092 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.872062    2575 generic.go:356] "Generic (PLEG): container finished" podID="62b1ce98-99da-4bc5-a8d7-d8e08af3111a" containerID="12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242" exitCode=0
Sep 16 12:50:07.872204 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.872102    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm"
Sep 16 12:50:07.872204 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.872150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm" event={"ID":"62b1ce98-99da-4bc5-a8d7-d8e08af3111a","Type":"ContainerDied","Data":"12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242"}
Sep 16 12:50:07.872204 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.872196    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fc58f76c4-4jhwm" event={"ID":"62b1ce98-99da-4bc5-a8d7-d8e08af3111a","Type":"ContainerDied","Data":"471c7181c02c49353aae0b8f1ee7369d4b1a1ef1b2d18f30528a496d55c7f7fe"}
Sep 16 12:50:07.872325 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.872216    2575 scope.go:117] "RemoveContainer" containerID="12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242"
Sep 16 12:50:07.880606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.880591    2575 scope.go:117] "RemoveContainer" containerID="12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242"
Sep 16 12:50:07.880929 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:50:07.880903    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242\": container with ID starting with 12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242 not found: ID does not exist" containerID="12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242"
Sep 16 12:50:07.881024 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.880934    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242"} err="failed to get container status \"12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242\": rpc error: code = NotFound desc = could not find container \"12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242\": container with ID starting with 12285a05088ffb1e44bd63d5d6266cbe9f00bf87fe509d9edeb810d86bd8c242 not found: ID does not exist"
Sep 16 12:50:07.896684 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.896662    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fc58f76c4-4jhwm"]
Sep 16 12:50:07.898872 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:07.898852    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fc58f76c4-4jhwm"]
Sep 16 12:50:08.481821 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:08.481789    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:08.582260 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:08.582235    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:09.116704 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:09.116679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:50:09.119066 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:09.119040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60d9fd9-2504-4b8d-b9a6-257b20027e36-metrics-certs\") pod \"network-metrics-daemon-6xf6c\" (UID: \"f60d9fd9-2504-4b8d-b9a6-257b20027e36\") " pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:50:09.327714 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:09.327680    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62b1ce98-99da-4bc5-a8d7-d8e08af3111a" path="/var/lib/kubelet/pods/62b1ce98-99da-4bc5-a8d7-d8e08af3111a/volumes"
Sep 16 12:50:09.351857 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:09.351835    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v25dx\""
Sep 16 12:50:09.359507 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:09.359490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xf6c"
Sep 16 12:50:09.625726 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:09.625633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:09.631306 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:09.631281    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xf6c"]
Sep 16 12:50:09.633820 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:50:09.633796    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60d9fd9_2504_4b8d_b9a6_257b20027e36.slice/crio-b97f3861e05496b417ab0f56fe51802e34472320731a627e4025603a6b8c413c WatchSource:0}: Error finding container b97f3861e05496b417ab0f56fe51802e34472320731a627e4025603a6b8c413c: Status 404 returned error can't find the container with id b97f3861e05496b417ab0f56fe51802e34472320731a627e4025603a6b8c413c
Sep 16 12:50:09.727924 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:09.727895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:09.881119 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:09.881054    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xf6c" event={"ID":"f60d9fd9-2504-4b8d-b9a6-257b20027e36","Type":"ContainerStarted","Data":"b97f3861e05496b417ab0f56fe51802e34472320731a627e4025603a6b8c413c"}
Sep 16 12:50:11.891191 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:11.891157    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xf6c" event={"ID":"f60d9fd9-2504-4b8d-b9a6-257b20027e36","Type":"ContainerStarted","Data":"11ed0d72e5f5b7a96b50a36d22b3791c9687a291ce3cdb7dee9e48051eb1a27a"}
Sep 16 12:50:11.891626 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:11.891210    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xf6c" event={"ID":"f60d9fd9-2504-4b8d-b9a6-257b20027e36","Type":"ContainerStarted","Data":"e4c2bfbea74e76ae33b04567ac4f452ae8d3dfa52227deb905d96288d2df8977"}
Sep 16 12:50:11.910182 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:11.910139    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6xf6c" podStartSLOduration=130.672471957 podStartE2EDuration="2m11.910125123s" totalImagesPullingTime="1.237653166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:50:11.90995404 +0000 UTC m=+133.091096693" watchObservedRunningTime="2026-09-16 12:50:11.910125123 +0000 UTC m=+133.091267776"
Sep 16 12:50:11.977789 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:11.977768    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:11.977925 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:11.977814    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:11.982137 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:11.982119    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:12.010714 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:12.010691    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:12.085709 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:12.085688    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:12.899901 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:12.899873    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:50:13.095591 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:13.095556    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:13.197679 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:13.197571    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:13.297509 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:13.297483    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:14.157496 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:14.157461    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:14.222923 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:14.222891    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:18.901937 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:18.901906    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:18.975132 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:18.975106    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:19.926262 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:19.926235    2575 generic.go:356] "Generic (PLEG): container finished" podID="6ae74286-8945-465a-9d8b-5f3ed411bade" containerID="137d41cd2506d403f00563f663904aebcd4bb8f7e77baa3ff48c7a0ecde76de2" exitCode=0
Sep 16 12:50:19.926536 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:19.926295    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xt4lw" event={"ID":"6ae74286-8945-465a-9d8b-5f3ed411bade","Type":"ContainerDied","Data":"137d41cd2506d403f00563f663904aebcd4bb8f7e77baa3ff48c7a0ecde76de2"}
Sep 16 12:50:19.926586 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:19.926572    2575 scope.go:117] "RemoveContainer" containerID="137d41cd2506d403f00563f663904aebcd4bb8f7e77baa3ff48c7a0ecde76de2"
Sep 16 12:50:20.932007 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:20.931971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-xt4lw" event={"ID":"6ae74286-8945-465a-9d8b-5f3ed411bade","Type":"ContainerStarted","Data":"305e64cfe5cdc4cff849907a6041bd27d045ef7eb1f549fbaaed1b5dc8cb6514"}
Sep 16 12:50:20.961100 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:20.961067    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kr5tj_0db4d710-77d0-4884-a702-739868c1c6ad/serve-healthcheck-canary/0.log"
Sep 16 12:50:24.948134 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:24.948064    2575 generic.go:356] "Generic (PLEG): container finished" podID="daf832f1-d96a-415c-8eb3-d6302f20b521" containerID="1f95303699ad02899b3c524cb3ce4ebf904058c2762a05f6cbe91ffeb4890588" exitCode=0
Sep 16 12:50:24.948134 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:24.948122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh" event={"ID":"daf832f1-d96a-415c-8eb3-d6302f20b521","Type":"ContainerDied","Data":"1f95303699ad02899b3c524cb3ce4ebf904058c2762a05f6cbe91ffeb4890588"}
Sep 16 12:50:24.948469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:24.948452    2575 scope.go:117] "RemoveContainer" containerID="1f95303699ad02899b3c524cb3ce4ebf904058c2762a05f6cbe91ffeb4890588"
Sep 16 12:50:25.953065 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:25.953032    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-sfgdh" event={"ID":"daf832f1-d96a-415c-8eb3-d6302f20b521","Type":"ContainerStarted","Data":"4e3a8a2152fe81b4bb48a81a1a5909a4784b27e285064dca7deb5cc5daf3464f"}
Sep 16 12:50:26.250113 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:26.250034    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:26.318439 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:26.318411    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:26.428758 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:26.428723    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:26.527500 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:26.527404    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:26.619200 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:26.619165    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:33.897756 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:33.897660    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:50:33.953751 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:50:33.953718    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:13.795165 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:13.795130    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:14.292220 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:14.292189    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:21.932762 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:21.932729    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d6fc7ddc4-hjv7l"]
Sep 16 12:51:21.933117 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:21.932976    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b1ce98-99da-4bc5-a8d7-d8e08af3111a" containerName="registry"
Sep 16 12:51:21.933117 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:21.932987    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b1ce98-99da-4bc5-a8d7-d8e08af3111a" containerName="registry"
Sep 16 12:51:21.933117 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:21.933041    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="62b1ce98-99da-4bc5-a8d7-d8e08af3111a" containerName="registry"
Sep 16 12:51:21.935814 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:21.935798    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:21.946350 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:21.946325    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d6fc7ddc4-hjv7l"]
Sep 16 12:51:21.986933 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:21.986914    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:22.027497 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.027473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-serving-cert\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.027578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.027516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-console-config\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.027578 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.027544    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-trusted-ca-bundle\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.027671 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.027582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46zj\" (UniqueName: \"kubernetes.io/projected/af864c98-a1de-43a8-8146-b6d46af84699-kube-api-access-g46zj\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.027671 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.027613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-service-ca\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.027671 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.027633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-oauth-config\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.027671 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.027664    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-oauth-serving-cert\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.128809 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.128781    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-service-ca\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.128898 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.128816    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-oauth-config\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.128898 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.128870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-oauth-serving-cert\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.128967 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.128914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-serving-cert\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.128967 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.128952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-console-config\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.129041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.128972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-trusted-ca-bundle\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.129041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.128995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46zj\" (UniqueName: \"kubernetes.io/projected/af864c98-a1de-43a8-8146-b6d46af84699-kube-api-access-g46zj\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.129946 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.129865    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-oauth-serving-cert\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.130196 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.130171    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-console-config\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.130350 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.130124    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-trusted-ca-bundle\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.130459 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.129955    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-service-ca\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.132247 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.132218    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-oauth-config\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.132394 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.132368    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-serving-cert\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.138224 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.138202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46zj\" (UniqueName: \"kubernetes.io/projected/af864c98-a1de-43a8-8146-b6d46af84699-kube-api-access-g46zj\") pod \"console-6d6fc7ddc4-hjv7l\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") " pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.244662 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.244575    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:22.515819 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:22.515743    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d6fc7ddc4-hjv7l"]
Sep 16 12:51:22.518872 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:51:22.518845    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf864c98_a1de_43a8_8146_b6d46af84699.slice/crio-c01390e760b663d461117fcf39af23132c8ba83a39b489aafd209e0ce3780244 WatchSource:0}: Error finding container c01390e760b663d461117fcf39af23132c8ba83a39b489aafd209e0ce3780244: Status 404 returned error can't find the container with id c01390e760b663d461117fcf39af23132c8ba83a39b489aafd209e0ce3780244
Sep 16 12:51:23.156981 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:23.156953    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6fc7ddc4-hjv7l" event={"ID":"af864c98-a1de-43a8-8146-b6d46af84699","Type":"ContainerStarted","Data":"4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b"}
Sep 16 12:51:23.157312 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:23.156995    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6fc7ddc4-hjv7l" event={"ID":"af864c98-a1de-43a8-8146-b6d46af84699","Type":"ContainerStarted","Data":"c01390e760b663d461117fcf39af23132c8ba83a39b489aafd209e0ce3780244"}
Sep 16 12:51:23.176412 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:23.176366    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d6fc7ddc4-hjv7l" podStartSLOduration=2.176351606 podStartE2EDuration="2.176351606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:23.175574877 +0000 UTC m=+204.356717531" watchObservedRunningTime="2026-09-16 12:51:23.176351606 +0000 UTC m=+204.357494259"
Sep 16 12:51:29.016887 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:29.016850    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:29.075815 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:29.075782    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:32.245502 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:32.245467    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:32.245948 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:32.245539    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:32.250331 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:32.250312    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:33.195905 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:33.195879    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:51:33.249036 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:33.249006    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786db67b48-dr6sg"]
Sep 16 12:51:33.401786 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:33.401749    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:33.495067 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:33.495001    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:33.702566 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:33.702535    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:33.763544 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:33.763480    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:34.303166 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:34.303132    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:35.306546 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:35.306517    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:35.486167 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:35.486137    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:40.746117 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:40.746077    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:40.990197 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:40.990172    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:41.065991 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:41.065917    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:44.461371 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:44.461337    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d6fc7ddc4-hjv7l"]
Sep 16 12:51:44.594427 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:44.594393    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:44.696166 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:44.696139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:44.757412 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:44.757349    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:48.616792 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:48.616761    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:48.968966 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:48.968896    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:49.059839 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:49.059813    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:51.000366 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:51.000334    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:51.078926 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:51.078899    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:51:58.268516 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.268479    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-786db67b48-dr6sg" podUID="01431672-4090-4d29-9116-3bf749f80aee" containerName="console" containerID="cri-o://d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121" gracePeriod=15
Sep 16 12:51:58.505815 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.505788    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:51:58.578591 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.578517    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-serving-cert\") pod \"01431672-4090-4d29-9116-3bf749f80aee\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") "
Sep 16 12:51:58.578591 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.578579    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-oauth-serving-cert\") pod \"01431672-4090-4d29-9116-3bf749f80aee\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") "
Sep 16 12:51:58.578794 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.578632    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-trusted-ca-bundle\") pod \"01431672-4090-4d29-9116-3bf749f80aee\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") "
Sep 16 12:51:58.578794 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.578680    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-console-config\") pod \"01431672-4090-4d29-9116-3bf749f80aee\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") "
Sep 16 12:51:58.578794 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.578717    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-service-ca\") pod \"01431672-4090-4d29-9116-3bf749f80aee\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") "
Sep 16 12:51:58.578794 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.578749    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-oauth-config\") pod \"01431672-4090-4d29-9116-3bf749f80aee\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") "
Sep 16 12:51:58.578982 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.578794    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vm9x\" (UniqueName: \"kubernetes.io/projected/01431672-4090-4d29-9116-3bf749f80aee-kube-api-access-2vm9x\") pod \"01431672-4090-4d29-9116-3bf749f80aee\" (UID: \"01431672-4090-4d29-9116-3bf749f80aee\") "
Sep 16 12:51:58.579089 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.579059    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-console-config" (OuterVolumeSpecName: "console-config") pod "01431672-4090-4d29-9116-3bf749f80aee" (UID: "01431672-4090-4d29-9116-3bf749f80aee"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:58.579089 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.579072    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "01431672-4090-4d29-9116-3bf749f80aee" (UID: "01431672-4090-4d29-9116-3bf749f80aee"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:58.579207 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.579092    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "01431672-4090-4d29-9116-3bf749f80aee" (UID: "01431672-4090-4d29-9116-3bf749f80aee"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:58.579207 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.579121    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-service-ca" (OuterVolumeSpecName: "service-ca") pod "01431672-4090-4d29-9116-3bf749f80aee" (UID: "01431672-4090-4d29-9116-3bf749f80aee"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:51:58.580928 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.580901    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "01431672-4090-4d29-9116-3bf749f80aee" (UID: "01431672-4090-4d29-9116-3bf749f80aee"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:58.581093 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.581068    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01431672-4090-4d29-9116-3bf749f80aee-kube-api-access-2vm9x" (OuterVolumeSpecName: "kube-api-access-2vm9x") pod "01431672-4090-4d29-9116-3bf749f80aee" (UID: "01431672-4090-4d29-9116-3bf749f80aee"). InnerVolumeSpecName "kube-api-access-2vm9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:51:58.581303 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.581282    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "01431672-4090-4d29-9116-3bf749f80aee" (UID: "01431672-4090-4d29-9116-3bf749f80aee"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:51:58.679882 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.679849    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-serving-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:51:58.679882 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.679879    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-oauth-serving-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:51:58.679882 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.679889    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-trusted-ca-bundle\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:51:58.680041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.679898    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-console-config\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:51:58.680041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.679906    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01431672-4090-4d29-9116-3bf749f80aee-service-ca\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:51:58.680041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.679914    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01431672-4090-4d29-9116-3bf749f80aee-console-oauth-config\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:51:58.680041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:58.679923    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vm9x\" (UniqueName: \"kubernetes.io/projected/01431672-4090-4d29-9116-3bf749f80aee-kube-api-access-2vm9x\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:51:59.287055 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.287020    2575 generic.go:356] "Generic (PLEG): container finished" podID="01431672-4090-4d29-9116-3bf749f80aee" containerID="d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121" exitCode=0
Sep 16 12:51:59.287407 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.287064    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786db67b48-dr6sg" event={"ID":"01431672-4090-4d29-9116-3bf749f80aee","Type":"ContainerDied","Data":"d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121"}
Sep 16 12:51:59.287407 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.287071    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786db67b48-dr6sg"
Sep 16 12:51:59.287407 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.287129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786db67b48-dr6sg" event={"ID":"01431672-4090-4d29-9116-3bf749f80aee","Type":"ContainerDied","Data":"3db155fe041998338d78f99d94bc3058934876e87335905c7078f26d99d96cf4"}
Sep 16 12:51:59.287407 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.287149    2575 scope.go:117] "RemoveContainer" containerID="d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121"
Sep 16 12:51:59.295323 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.295308    2575 scope.go:117] "RemoveContainer" containerID="d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121"
Sep 16 12:51:59.295590 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:51:59.295572    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121\": container with ID starting with d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121 not found: ID does not exist" containerID="d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121"
Sep 16 12:51:59.295632 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.295599    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121"} err="failed to get container status \"d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121\": rpc error: code = NotFound desc = could not find container \"d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121\": container with ID starting with d7da7425d453fd73149e9b11bc74eb5a99a78c9c0e1b61a28a96285f7ab61121 not found: ID does not exist"
Sep 16 12:51:59.307691 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.307668    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786db67b48-dr6sg"]
Sep 16 12:51:59.309183 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.309162    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-786db67b48-dr6sg"]
Sep 16 12:51:59.327633 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:51:59.327613    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01431672-4090-4d29-9116-3bf749f80aee" path="/var/lib/kubelet/pods/01431672-4090-4d29-9116-3bf749f80aee/volumes"
Sep 16 12:52:09.485196 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.485161    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d6fc7ddc4-hjv7l" podUID="af864c98-a1de-43a8-8146-b6d46af84699" containerName="console" containerID="cri-o://4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b" gracePeriod=15
Sep 16 12:52:09.719022 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.718998    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:52:09.753562 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.753507    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-trusted-ca-bundle\") pod \"af864c98-a1de-43a8-8146-b6d46af84699\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") "
Sep 16 12:52:09.753562 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.753540    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-service-ca\") pod \"af864c98-a1de-43a8-8146-b6d46af84699\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") "
Sep 16 12:52:09.753748 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.753566    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-serving-cert\") pod \"af864c98-a1de-43a8-8146-b6d46af84699\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") "
Sep 16 12:52:09.753748 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.753598    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-oauth-serving-cert\") pod \"af864c98-a1de-43a8-8146-b6d46af84699\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") "
Sep 16 12:52:09.753748 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.753625    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-console-config\") pod \"af864c98-a1de-43a8-8146-b6d46af84699\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") "
Sep 16 12:52:09.753748 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.753663    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g46zj\" (UniqueName: \"kubernetes.io/projected/af864c98-a1de-43a8-8146-b6d46af84699-kube-api-access-g46zj\") pod \"af864c98-a1de-43a8-8146-b6d46af84699\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") "
Sep 16 12:52:09.753748 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.753690    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-oauth-config\") pod \"af864c98-a1de-43a8-8146-b6d46af84699\" (UID: \"af864c98-a1de-43a8-8146-b6d46af84699\") "
Sep 16 12:52:09.754024 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.753997    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-service-ca" (OuterVolumeSpecName: "service-ca") pod "af864c98-a1de-43a8-8146-b6d46af84699" (UID: "af864c98-a1de-43a8-8146-b6d46af84699"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:52:09.754086 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.754026    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "af864c98-a1de-43a8-8146-b6d46af84699" (UID: "af864c98-a1de-43a8-8146-b6d46af84699"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:52:09.754141 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.754117    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "af864c98-a1de-43a8-8146-b6d46af84699" (UID: "af864c98-a1de-43a8-8146-b6d46af84699"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:52:09.754181 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.754128    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-console-config" (OuterVolumeSpecName: "console-config") pod "af864c98-a1de-43a8-8146-b6d46af84699" (UID: "af864c98-a1de-43a8-8146-b6d46af84699"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:52:09.755967 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.755940    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "af864c98-a1de-43a8-8146-b6d46af84699" (UID: "af864c98-a1de-43a8-8146-b6d46af84699"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:52:09.756063 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.755968    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af864c98-a1de-43a8-8146-b6d46af84699-kube-api-access-g46zj" (OuterVolumeSpecName: "kube-api-access-g46zj") pod "af864c98-a1de-43a8-8146-b6d46af84699" (UID: "af864c98-a1de-43a8-8146-b6d46af84699"). InnerVolumeSpecName "kube-api-access-g46zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:52:09.756063 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.756047    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "af864c98-a1de-43a8-8146-b6d46af84699" (UID: "af864c98-a1de-43a8-8146-b6d46af84699"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:52:09.854968 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.854944    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-oauth-serving-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:52:09.854968 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.854963    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-console-config\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:52:09.854968 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.854972    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g46zj\" (UniqueName: \"kubernetes.io/projected/af864c98-a1de-43a8-8146-b6d46af84699-kube-api-access-g46zj\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:52:09.855132 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.854982    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-oauth-config\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:52:09.855132 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.854989    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-trusted-ca-bundle\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:52:09.855132 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.855016    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af864c98-a1de-43a8-8146-b6d46af84699-service-ca\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:52:09.855132 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:09.855025    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af864c98-a1de-43a8-8146-b6d46af84699-console-serving-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:52:10.329066 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:10.329032    2575 generic.go:356] "Generic (PLEG): container finished" podID="af864c98-a1de-43a8-8146-b6d46af84699" containerID="4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b" exitCode=0
Sep 16 12:52:10.329212 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:10.329071    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6fc7ddc4-hjv7l" event={"ID":"af864c98-a1de-43a8-8146-b6d46af84699","Type":"ContainerDied","Data":"4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b"}
Sep 16 12:52:10.329212 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:10.329081    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6fc7ddc4-hjv7l"
Sep 16 12:52:10.329212 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:10.329106    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6fc7ddc4-hjv7l" event={"ID":"af864c98-a1de-43a8-8146-b6d46af84699","Type":"ContainerDied","Data":"c01390e760b663d461117fcf39af23132c8ba83a39b489aafd209e0ce3780244"}
Sep 16 12:52:10.329212 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:10.329129    2575 scope.go:117] "RemoveContainer" containerID="4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b"
Sep 16 12:52:10.338132 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:10.338115    2575 scope.go:117] "RemoveContainer" containerID="4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b"
Sep 16 12:52:10.338394 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:52:10.338375    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b\": container with ID starting with 4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b not found: ID does not exist" containerID="4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b"
Sep 16 12:52:10.338451 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:10.338406    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b"} err="failed to get container status \"4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b\": rpc error: code = NotFound desc = could not find container \"4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b\": container with ID starting with 4b5a3e50e730e11787e67c98d83c6e1dd443d3251897c4ff72670cb650366b8b not found: ID does not exist"
Sep 16 12:52:10.349421 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:10.349392    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d6fc7ddc4-hjv7l"]
Sep 16 12:52:10.353297 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:10.353278    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d6fc7ddc4-hjv7l"]
Sep 16 12:52:11.326830 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:11.326801    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af864c98-a1de-43a8-8146-b6d46af84699" path="/var/lib/kubelet/pods/af864c98-a1de-43a8-8146-b6d46af84699/volumes"
Sep 16 12:52:37.248634 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:37.248602    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:52:53.872777 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:53.872745    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:52:53.943963 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:53.943931    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:52:59.237169 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:52:59.237143    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:53:32.791056 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:32.791021    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:53:32.882575 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:32.882545    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:53:32.948255 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:32.948229    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:53:33.958995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.958965    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"]
Sep 16 12:53:33.959339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.959223    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af864c98-a1de-43a8-8146-b6d46af84699" containerName="console"
Sep 16 12:53:33.959339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.959234    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="af864c98-a1de-43a8-8146-b6d46af84699" containerName="console"
Sep 16 12:53:33.959339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.959254    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01431672-4090-4d29-9116-3bf749f80aee" containerName="console"
Sep 16 12:53:33.959339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.959260    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="01431672-4090-4d29-9116-3bf749f80aee" containerName="console"
Sep 16 12:53:33.959339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.959313    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="01431672-4090-4d29-9116-3bf749f80aee" containerName="console"
Sep 16 12:53:33.959339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.959324    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="af864c98-a1de-43a8-8146-b6d46af84699" containerName="console"
Sep 16 12:53:33.962312 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.962297    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:33.964916 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.964896    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:53:33.966224 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.965316    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p27d8\""
Sep 16 12:53:33.970035 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.970010    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:53:33.972032 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:33.972005    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"]
Sep 16 12:53:34.001345 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.001316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.001453 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.001391    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.001453 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.001429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdlq\" (UniqueName: \"kubernetes.io/projected/9d9f3001-8514-47dc-a7b5-55200a26b199-kube-api-access-5jdlq\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.102110 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.102086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdlq\" (UniqueName: \"kubernetes.io/projected/9d9f3001-8514-47dc-a7b5-55200a26b199-kube-api-access-5jdlq\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.102214 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.102126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.102214 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.102199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.102502 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.102484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.102583 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.102515    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.111130 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.111109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdlq\" (UniqueName: \"kubernetes.io/projected/9d9f3001-8514-47dc-a7b5-55200a26b199-kube-api-access-5jdlq\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.276452 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.276392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:34.546244 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.546181    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"]
Sep 16 12:53:34.549166 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:53:34.549146    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9f3001_8514_47dc_a7b5_55200a26b199.slice/crio-240655e9bf9a98058ff1b90bbe77fde6382985478737038b9eeb5fc6f6bed3bf WatchSource:0}: Error finding container 240655e9bf9a98058ff1b90bbe77fde6382985478737038b9eeb5fc6f6bed3bf: Status 404 returned error can't find the container with id 240655e9bf9a98058ff1b90bbe77fde6382985478737038b9eeb5fc6f6bed3bf
Sep 16 12:53:34.551102 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.551087    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:53:34.618442 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:34.618413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2" event={"ID":"9d9f3001-8514-47dc-a7b5-55200a26b199","Type":"ContainerStarted","Data":"240655e9bf9a98058ff1b90bbe77fde6382985478737038b9eeb5fc6f6bed3bf"}
Sep 16 12:53:41.647535 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:41.647504    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerID="86e9aa278634ce54377adf95fb17a6c4e50c3c6e7369a35f311fe86dc7ab216d" exitCode=0
Sep 16 12:53:41.647937 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:41.647577    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2" event={"ID":"9d9f3001-8514-47dc-a7b5-55200a26b199","Type":"ContainerDied","Data":"86e9aa278634ce54377adf95fb17a6c4e50c3c6e7369a35f311fe86dc7ab216d"}
Sep 16 12:53:43.656289 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:43.656262    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerID="8f75261ddaf005088b08bf9547a2e53a14f2595418c9d17a91377a4089db04c8" exitCode=0
Sep 16 12:53:43.656613 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:43.656343    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2" event={"ID":"9d9f3001-8514-47dc-a7b5-55200a26b199","Type":"ContainerDied","Data":"8f75261ddaf005088b08bf9547a2e53a14f2595418c9d17a91377a4089db04c8"}
Sep 16 12:53:50.685365 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:50.685336    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerID="316c3bf091c349177a8952b3b8f3558b35ebd79b9a6a05e7a30899c189ee4d19" exitCode=0
Sep 16 12:53:50.685685 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:50.685381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2" event={"ID":"9d9f3001-8514-47dc-a7b5-55200a26b199","Type":"ContainerDied","Data":"316c3bf091c349177a8952b3b8f3558b35ebd79b9a6a05e7a30899c189ee4d19"}
Sep 16 12:53:51.808218 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.808195    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:51.834281 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.834254    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-bundle\") pod \"9d9f3001-8514-47dc-a7b5-55200a26b199\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") "
Sep 16 12:53:51.834384 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.834295    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jdlq\" (UniqueName: \"kubernetes.io/projected/9d9f3001-8514-47dc-a7b5-55200a26b199-kube-api-access-5jdlq\") pod \"9d9f3001-8514-47dc-a7b5-55200a26b199\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") "
Sep 16 12:53:51.834384 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.834329    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-util\") pod \"9d9f3001-8514-47dc-a7b5-55200a26b199\" (UID: \"9d9f3001-8514-47dc-a7b5-55200a26b199\") "
Sep 16 12:53:51.835105 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.835017    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-bundle" (OuterVolumeSpecName: "bundle") pod "9d9f3001-8514-47dc-a7b5-55200a26b199" (UID: "9d9f3001-8514-47dc-a7b5-55200a26b199"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:53:51.836623 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.836595    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9f3001-8514-47dc-a7b5-55200a26b199-kube-api-access-5jdlq" (OuterVolumeSpecName: "kube-api-access-5jdlq") pod "9d9f3001-8514-47dc-a7b5-55200a26b199" (UID: "9d9f3001-8514-47dc-a7b5-55200a26b199"). InnerVolumeSpecName "kube-api-access-5jdlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:53:51.838492 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.838467    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-util" (OuterVolumeSpecName: "util") pod "9d9f3001-8514-47dc-a7b5-55200a26b199" (UID: "9d9f3001-8514-47dc-a7b5-55200a26b199"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:53:51.934914 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.934893    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-bundle\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:53:51.934914 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.934914    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdlq\" (UniqueName: \"kubernetes.io/projected/9d9f3001-8514-47dc-a7b5-55200a26b199-kube-api-access-5jdlq\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:53:51.935025 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:51.934924    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d9f3001-8514-47dc-a7b5-55200a26b199-util\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:53:52.694233 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:52.694205    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2"
Sep 16 12:53:52.694399 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:52.694205    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ad22n2" event={"ID":"9d9f3001-8514-47dc-a7b5-55200a26b199","Type":"ContainerDied","Data":"240655e9bf9a98058ff1b90bbe77fde6382985478737038b9eeb5fc6f6bed3bf"}
Sep 16 12:53:52.694399 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:52.694325    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="240655e9bf9a98058ff1b90bbe77fde6382985478737038b9eeb5fc6f6bed3bf"
Sep 16 12:53:55.123357 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:55.123325    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:53:56.431032 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.430999    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:53:56.603597 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.603567    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"]
Sep 16 12:53:56.603857 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.603845    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerName="util"
Sep 16 12:53:56.603899 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.603858    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerName="util"
Sep 16 12:53:56.603899 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.603874    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerName="pull"
Sep 16 12:53:56.603899 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.603879    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerName="pull"
Sep 16 12:53:56.603899 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.603887    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerName="extract"
Sep 16 12:53:56.603899 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.603893    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerName="extract"
Sep 16 12:53:56.604040 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.603940    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d9f3001-8514-47dc-a7b5-55200a26b199" containerName="extract"
Sep 16 12:53:56.606997 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.606981    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:53:56.611417 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.611398    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 12:53:56.611760 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.611745    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-jjhh2\""
Sep 16 12:53:56.612737 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.612713    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 12:53:56.612840 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.612743    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 12:53:56.623613 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.623586    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"]
Sep 16 12:53:56.672045 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.672024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c1796915-4366-4365-99a0-c472f44f2b2a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gx2p4\" (UID: \"c1796915-4366-4365-99a0-c472f44f2b2a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:53:56.672157 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.672076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szntl\" (UniqueName: \"kubernetes.io/projected/c1796915-4366-4365-99a0-c472f44f2b2a-kube-api-access-szntl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gx2p4\" (UID: \"c1796915-4366-4365-99a0-c472f44f2b2a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:53:56.773009 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.772928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szntl\" (UniqueName: \"kubernetes.io/projected/c1796915-4366-4365-99a0-c472f44f2b2a-kube-api-access-szntl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gx2p4\" (UID: \"c1796915-4366-4365-99a0-c472f44f2b2a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:53:56.773009 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.772976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c1796915-4366-4365-99a0-c472f44f2b2a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gx2p4\" (UID: \"c1796915-4366-4365-99a0-c472f44f2b2a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:53:56.775488 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.775466    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/c1796915-4366-4365-99a0-c472f44f2b2a-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gx2p4\" (UID: \"c1796915-4366-4365-99a0-c472f44f2b2a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:53:56.782538 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.782500    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szntl\" (UniqueName: \"kubernetes.io/projected/c1796915-4366-4365-99a0-c472f44f2b2a-kube-api-access-szntl\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-gx2p4\" (UID: \"c1796915-4366-4365-99a0-c472f44f2b2a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:53:56.916452 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:56.916404    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:53:57.190722 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:57.190694    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"]
Sep 16 12:53:57.193334 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:53:57.193298    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1796915_4366_4365_99a0_c472f44f2b2a.slice/crio-d8e282293fbb474cab2cd20947083f0d63fe62ab3beb30802a93de7ac5893a13 WatchSource:0}: Error finding container d8e282293fbb474cab2cd20947083f0d63fe62ab3beb30802a93de7ac5893a13: Status 404 returned error can't find the container with id d8e282293fbb474cab2cd20947083f0d63fe62ab3beb30802a93de7ac5893a13
Sep 16 12:53:57.717469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:53:57.717422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4" event={"ID":"c1796915-4366-4365-99a0-c472f44f2b2a","Type":"ContainerStarted","Data":"d8e282293fbb474cab2cd20947083f0d63fe62ab3beb30802a93de7ac5893a13"}
Sep 16 12:54:00.151397 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.151361    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:00.428143 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.428071    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:00.432540 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.432519    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-ptx4s"]
Sep 16 12:54:00.435607 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.435591    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:00.438145 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.438125    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 12:54:00.438227 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.438126    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6tncr\""
Sep 16 12:54:00.438227 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.438185    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 12:54:00.445566 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.445546    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-ptx4s"]
Sep 16 12:54:00.501120 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.501092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8msdr\" (UniqueName: \"kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-kube-api-access-8msdr\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:00.501250 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.501149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ae2778a4-62d1-4487-92b3-214cc79fcef6-cabundle0\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:00.501250 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.501175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-certificates\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:00.602087 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.602054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ae2778a4-62d1-4487-92b3-214cc79fcef6-cabundle0\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:00.602087 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.602087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-certificates\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:00.602326 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.602127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8msdr\" (UniqueName: \"kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-kube-api-access-8msdr\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:00.602326 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:00.602224    2575 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 12:54:00.602326 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:00.602262    2575 secret.go:281] references non-existent secret key: ca.crt
Sep 16 12:54:00.602326 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:00.602274    2575 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 12:54:00.602326 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:00.602290    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-ptx4s: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 12:54:00.602559 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:00.602357    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-certificates podName:ae2778a4-62d1-4487-92b3-214cc79fcef6 nodeName:}" failed. No retries permitted until 2026-09-16 12:54:01.102336973 +0000 UTC m=+362.283479607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-certificates") pod "keda-operator-579c8958f7-ptx4s" (UID: "ae2778a4-62d1-4487-92b3-214cc79fcef6") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 12:54:00.602756 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.602739    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ae2778a4-62d1-4487-92b3-214cc79fcef6-cabundle0\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:00.612534 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.612510    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8msdr\" (UniqueName: \"kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-kube-api-access-8msdr\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:00.689574 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.689496    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:00.731011 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.730980    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4" event={"ID":"c1796915-4366-4365-99a0-c472f44f2b2a","Type":"ContainerStarted","Data":"c9bdf0cd86edaec1d19923846ffb065f68bb4b3d98f0163452c83113c106fe54"}
Sep 16 12:54:00.731184 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.731127    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:54:00.752234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.752189    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4" podStartSLOduration=2.215129515 podStartE2EDuration="4.752173065s" totalImagesPullingTime="2.53704355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:00.75069004 +0000 UTC m=+361.931832687" watchObservedRunningTime="2026-09-16 12:54:00.752173065 +0000 UTC m=+361.933315717"
Sep 16 12:54:00.812118 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.812091    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:00.888321 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.888294    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:00.889215 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.889197    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"]
Sep 16 12:54:00.892367 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.892353    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:00.894958 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.894938    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 12:54:00.902267 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:00.902240    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"]
Sep 16 12:54:01.004496 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.004422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldmzw\" (UniqueName: \"kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-kube-api-access-ldmzw\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:01.004609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.004532    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:01.004609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.004554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d6bc6300-f7e6-4df2-87da-c05869d2783c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:01.105865 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.105841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldmzw\" (UniqueName: \"kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-kube-api-access-ldmzw\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:01.105971 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.105954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-certificates\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:01.106007 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.105994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:01.106043 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.106015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d6bc6300-f7e6-4df2-87da-c05869d2783c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:01.106105 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.106087    2575 secret.go:281] references non-existent secret key: ca.crt
Sep 16 12:54:01.106147 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.106104    2575 secret.go:281] references non-existent secret key: tls.crt
Sep 16 12:54:01.106147 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.106109    2575 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 12:54:01.106147 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.106119    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-ptx4s: references non-existent secret key: ca.crt
Sep 16 12:54:01.106147 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.106121    2575 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 16 12:54:01.106147 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.106141    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg: references non-existent secret key: tls.crt
Sep 16 12:54:01.106309 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.106162    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-certificates podName:ae2778a4-62d1-4487-92b3-214cc79fcef6 nodeName:}" failed. No retries permitted until 2026-09-16 12:54:02.106149224 +0000 UTC m=+363.287291855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-certificates") pod "keda-operator-579c8958f7-ptx4s" (UID: "ae2778a4-62d1-4487-92b3-214cc79fcef6") : references non-existent secret key: ca.crt
Sep 16 12:54:01.106309 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.106196    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-certificates podName:d6bc6300-f7e6-4df2-87da-c05869d2783c nodeName:}" failed. No retries permitted until 2026-09-16 12:54:01.606180816 +0000 UTC m=+362.787323449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-certificates") pod "keda-metrics-apiserver-77857b7b98-9lczg" (UID: "d6bc6300-f7e6-4df2-87da-c05869d2783c") : references non-existent secret key: tls.crt
Sep 16 12:54:01.106411 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.106398    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d6bc6300-f7e6-4df2-87da-c05869d2783c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:01.116624 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.116601    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldmzw\" (UniqueName: \"kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-kube-api-access-ldmzw\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:01.610948 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.610909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:01.611336 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.611058    2575 secret.go:281] references non-existent secret key: tls.crt
Sep 16 12:54:01.611336 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.611079    2575 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 16 12:54:01.611336 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.611097    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg: references non-existent secret key: tls.crt
Sep 16 12:54:01.611336 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.611159    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-certificates podName:d6bc6300-f7e6-4df2-87da-c05869d2783c nodeName:}" failed. No retries permitted until 2026-09-16 12:54:02.61114445 +0000 UTC m=+363.792287085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-certificates") pod "keda-metrics-apiserver-77857b7b98-9lczg" (UID: "d6bc6300-f7e6-4df2-87da-c05869d2783c") : references non-existent secret key: tls.crt
Sep 16 12:54:01.700396 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.700363    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:01.709963 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.709941    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-q22t4"]
Sep 16 12:54:01.713135 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.713119    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:01.716522 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.716498    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 12:54:01.724029 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.724007    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-q22t4"]
Sep 16 12:54:01.795914 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.795882    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:01.812359 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.812328    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vwm\" (UniqueName: \"kubernetes.io/projected/4026290a-2fcf-4b53-91fc-97913aacd153-kube-api-access-88vwm\") pod \"keda-admission-6c4d864b86-q22t4\" (UID: \"4026290a-2fcf-4b53-91fc-97913aacd153\") " pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:01.812469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.812458    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4026290a-2fcf-4b53-91fc-97913aacd153-certificates\") pod \"keda-admission-6c4d864b86-q22t4\" (UID: \"4026290a-2fcf-4b53-91fc-97913aacd153\") " pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:01.901239 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.901153    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:01.913303 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.913277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vwm\" (UniqueName: \"kubernetes.io/projected/4026290a-2fcf-4b53-91fc-97913aacd153-kube-api-access-88vwm\") pod \"keda-admission-6c4d864b86-q22t4\" (UID: \"4026290a-2fcf-4b53-91fc-97913aacd153\") " pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:01.913426 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.913339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4026290a-2fcf-4b53-91fc-97913aacd153-certificates\") pod \"keda-admission-6c4d864b86-q22t4\" (UID: \"4026290a-2fcf-4b53-91fc-97913aacd153\") " pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:01.913475 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.913433    2575 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 16 12:54:01.913475 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.913449    2575 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-q22t4: secret "keda-admission-webhooks-certs" not found
Sep 16 12:54:01.913553 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:54:01.913493    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4026290a-2fcf-4b53-91fc-97913aacd153-certificates podName:4026290a-2fcf-4b53-91fc-97913aacd153 nodeName:}" failed. No retries permitted until 2026-09-16 12:54:02.413480706 +0000 UTC m=+363.594623337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4026290a-2fcf-4b53-91fc-97913aacd153-certificates") pod "keda-admission-6c4d864b86-q22t4" (UID: "4026290a-2fcf-4b53-91fc-97913aacd153") : secret "keda-admission-webhooks-certs" not found
Sep 16 12:54:01.929327 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:01.929297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vwm\" (UniqueName: \"kubernetes.io/projected/4026290a-2fcf-4b53-91fc-97913aacd153-kube-api-access-88vwm\") pod \"keda-admission-6c4d864b86-q22t4\" (UID: \"4026290a-2fcf-4b53-91fc-97913aacd153\") " pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:02.096840 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.096807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:02.114718 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.114687    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-certificates\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:02.117207 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.117181    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ae2778a4-62d1-4487-92b3-214cc79fcef6-certificates\") pod \"keda-operator-579c8958f7-ptx4s\" (UID: \"ae2778a4-62d1-4487-92b3-214cc79fcef6\") " pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:02.197133 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.197044    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:02.244696 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.244634    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:02.297097 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.297066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:02.397057 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.397027    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:02.417434 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.417404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4026290a-2fcf-4b53-91fc-97913aacd153-certificates\") pod \"keda-admission-6c4d864b86-q22t4\" (UID: \"4026290a-2fcf-4b53-91fc-97913aacd153\") " pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:02.419961 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.419933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4026290a-2fcf-4b53-91fc-97913aacd153-certificates\") pod \"keda-admission-6c4d864b86-q22t4\" (UID: \"4026290a-2fcf-4b53-91fc-97913aacd153\") " pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:02.504544 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.504468    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:02.515728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.515707    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-ptx4s"]
Sep 16 12:54:02.517754 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:54:02.517717    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2778a4_62d1_4487_92b3_214cc79fcef6.slice/crio-1155ad60ae8ed7987c0d15f8eefa70d89d9d9e0fca6304dc580c3aaafbc3f0b2 WatchSource:0}: Error finding container 1155ad60ae8ed7987c0d15f8eefa70d89d9d9e0fca6304dc580c3aaafbc3f0b2: Status 404 returned error can't find the container with id 1155ad60ae8ed7987c0d15f8eefa70d89d9d9e0fca6304dc580c3aaafbc3f0b2
Sep 16 12:54:02.618116 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.618087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:02.620673 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.620625    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d6bc6300-f7e6-4df2-87da-c05869d2783c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9lczg\" (UID: \"d6bc6300-f7e6-4df2-87da-c05869d2783c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:02.623361 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.623344    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:02.702577 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.702549    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:02.740826 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.740797    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-ptx4s" event={"ID":"ae2778a4-62d1-4487-92b3-214cc79fcef6","Type":"ContainerStarted","Data":"1155ad60ae8ed7987c0d15f8eefa70d89d9d9e0fca6304dc580c3aaafbc3f0b2"}
Sep 16 12:54:02.893693 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.893661    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-q22t4"]
Sep 16 12:54:02.896675 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:54:02.896632    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4026290a_2fcf_4b53_91fc_97913aacd153.slice/crio-67c446dc81a3ac3618b42e23cb74cdf52f978edf277d15a5f5f5846e61171a5f WatchSource:0}: Error finding container 67c446dc81a3ac3618b42e23cb74cdf52f978edf277d15a5f5f5846e61171a5f: Status 404 returned error can't find the container with id 67c446dc81a3ac3618b42e23cb74cdf52f978edf277d15a5f5f5846e61171a5f
Sep 16 12:54:02.974252 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:02.974225    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"]
Sep 16 12:54:02.976819 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:54:02.976784    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bc6300_f7e6_4df2_87da_c05869d2783c.slice/crio-184389f4f47ff995acf88cea98a9593c08578866cf0760e38daaca573de4d3b7 WatchSource:0}: Error finding container 184389f4f47ff995acf88cea98a9593c08578866cf0760e38daaca573de4d3b7: Status 404 returned error can't find the container with id 184389f4f47ff995acf88cea98a9593c08578866cf0760e38daaca573de4d3b7
Sep 16 12:54:03.747801 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:03.747765    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg" event={"ID":"d6bc6300-f7e6-4df2-87da-c05869d2783c","Type":"ContainerStarted","Data":"184389f4f47ff995acf88cea98a9593c08578866cf0760e38daaca573de4d3b7"}
Sep 16 12:54:03.749358 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:03.749322    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-q22t4" event={"ID":"4026290a-2fcf-4b53-91fc-97913aacd153","Type":"ContainerStarted","Data":"67c446dc81a3ac3618b42e23cb74cdf52f978edf277d15a5f5f5846e61171a5f"}
Sep 16 12:54:04.756492 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:04.756450    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-q22t4" event={"ID":"4026290a-2fcf-4b53-91fc-97913aacd153","Type":"ContainerStarted","Data":"8828212d79be28c482deda98daa66fbe0aba04682368a79aa4edf1453fc71bb1"}
Sep 16 12:54:04.756964 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:04.756542    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:04.774504 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:04.774449    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-q22t4" podStartSLOduration=2.426597223 podStartE2EDuration="3.774433161s" totalImagesPullingTime="1.347835938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:04.772591719 +0000 UTC m=+365.953734376" watchObservedRunningTime="2026-09-16 12:54:04.774433161 +0000 UTC m=+365.955575814"
Sep 16 12:54:06.767080 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:06.767049    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-ptx4s" event={"ID":"ae2778a4-62d1-4487-92b3-214cc79fcef6","Type":"ContainerStarted","Data":"c5a893eb7f36d0586741646b6fd34bc500c0d79e6d4c9f1099ea99640f2281f2"}
Sep 16 12:54:06.767489 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:06.767144    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:06.768949 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:06.768923    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg" event={"ID":"d6bc6300-f7e6-4df2-87da-c05869d2783c","Type":"ContainerStarted","Data":"cbca1991e47047f35655385d45929ed2f5291405a73c28b34deae433c5884213"}
Sep 16 12:54:06.769075 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:06.768981    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:06.787885 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:06.787841    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-ptx4s" podStartSLOduration=2.949937319 podStartE2EDuration="6.787826506s" totalImagesPullingTime="3.837889187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:06.786604712 +0000 UTC m=+367.967747366" watchObservedRunningTime="2026-09-16 12:54:06.787826506 +0000 UTC m=+367.968969159"
Sep 16 12:54:06.805196 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:06.805143    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg" podStartSLOduration=3.426283047 podStartE2EDuration="6.805124904s" totalImagesPullingTime="3.378841857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:06.804501108 +0000 UTC m=+367.985643775" watchObservedRunningTime="2026-09-16 12:54:06.805124904 +0000 UTC m=+367.986267558"
Sep 16 12:54:10.911374 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:10.911340    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:10.978984 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:10.978955    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:17.778408 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:17.778380    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9lczg"
Sep 16 12:54:21.737660 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:21.737613    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-gx2p4"
Sep 16 12:54:25.763871 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:25.763835    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-q22t4"
Sep 16 12:54:27.776098 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:27.776069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-ptx4s"
Sep 16 12:54:56.306274 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.306244    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:56.406741 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.406705    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:54:56.563769 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.563696    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"]
Sep 16 12:54:56.565919 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.565903    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.568690 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.568671    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:54:56.568805 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.568709    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:54:56.570074 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.570054    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p27d8\""
Sep 16 12:54:56.578573 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.578554    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"]
Sep 16 12:54:56.717611 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.717572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.717781 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.717620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.717781 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.717670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjsq\" (UniqueName: \"kubernetes.io/projected/bfae2484-2326-45bf-9c9a-fc96c4d9206e-kube-api-access-rcjsq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.818975 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.818908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.818975 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.818950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.819142 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.818978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjsq\" (UniqueName: \"kubernetes.io/projected/bfae2484-2326-45bf-9c9a-fc96c4d9206e-kube-api-access-rcjsq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.819271 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.819252    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.819310 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.819276    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.827266 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.827246    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjsq\" (UniqueName: \"kubernetes.io/projected/bfae2484-2326-45bf-9c9a-fc96c4d9206e-kube-api-access-rcjsq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:56.874845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:56.874821    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:54:57.147001 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:57.146935    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"]
Sep 16 12:54:57.149207 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:54:57.149177    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfae2484_2326_45bf_9c9a_fc96c4d9206e.slice/crio-9900044cf22e9f3fc44b30c4d36adc8bd2a8ec17badf089ac56da04193c62e79 WatchSource:0}: Error finding container 9900044cf22e9f3fc44b30c4d36adc8bd2a8ec17badf089ac56da04193c62e79: Status 404 returned error can't find the container with id 9900044cf22e9f3fc44b30c4d36adc8bd2a8ec17badf089ac56da04193c62e79
Sep 16 12:54:57.962585 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:57.962550    2575 generic.go:356] "Generic (PLEG): container finished" podID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerID="4434a933b2035a09d0d7434a22bc9935e8688a24a669c6e276f6f7267be327e7" exitCode=0
Sep 16 12:54:57.962963 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:57.962594    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr" event={"ID":"bfae2484-2326-45bf-9c9a-fc96c4d9206e","Type":"ContainerDied","Data":"4434a933b2035a09d0d7434a22bc9935e8688a24a669c6e276f6f7267be327e7"}
Sep 16 12:54:57.962963 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:54:57.962634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr" event={"ID":"bfae2484-2326-45bf-9c9a-fc96c4d9206e","Type":"ContainerStarted","Data":"9900044cf22e9f3fc44b30c4d36adc8bd2a8ec17badf089ac56da04193c62e79"}
Sep 16 12:55:00.136146 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:55:00.136120    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfae2484_2326_45bf_9c9a_fc96c4d9206e.slice/crio-conmon-6d61a9b0ca12cca89a6d65bcfae2f50524c47b7f33795b8172eaae0d116e67a6.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 12:55:00.976785 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:00.976756    2575 generic.go:356] "Generic (PLEG): container finished" podID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerID="6d61a9b0ca12cca89a6d65bcfae2f50524c47b7f33795b8172eaae0d116e67a6" exitCode=0
Sep 16 12:55:00.976958 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:00.976836    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr" event={"ID":"bfae2484-2326-45bf-9c9a-fc96c4d9206e","Type":"ContainerDied","Data":"6d61a9b0ca12cca89a6d65bcfae2f50524c47b7f33795b8172eaae0d116e67a6"}
Sep 16 12:55:01.982606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:01.982570    2575 generic.go:356] "Generic (PLEG): container finished" podID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerID="c1ce8fab9f4cfd19c577ef491357723f05a8e308167858ba58260a5e434388e6" exitCode=0
Sep 16 12:55:01.982999 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:01.982634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr" event={"ID":"bfae2484-2326-45bf-9c9a-fc96c4d9206e","Type":"ContainerDied","Data":"c1ce8fab9f4cfd19c577ef491357723f05a8e308167858ba58260a5e434388e6"}
Sep 16 12:55:03.114121 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.114094    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:55:03.170349 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.170314    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-util\") pod \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") "
Sep 16 12:55:03.170349 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.170355    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-bundle\") pod \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") "
Sep 16 12:55:03.170551 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.170394    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcjsq\" (UniqueName: \"kubernetes.io/projected/bfae2484-2326-45bf-9c9a-fc96c4d9206e-kube-api-access-rcjsq\") pod \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\" (UID: \"bfae2484-2326-45bf-9c9a-fc96c4d9206e\") "
Sep 16 12:55:03.171204 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.171036    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-bundle" (OuterVolumeSpecName: "bundle") pod "bfae2484-2326-45bf-9c9a-fc96c4d9206e" (UID: "bfae2484-2326-45bf-9c9a-fc96c4d9206e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:55:03.172599 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.172566    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfae2484-2326-45bf-9c9a-fc96c4d9206e-kube-api-access-rcjsq" (OuterVolumeSpecName: "kube-api-access-rcjsq") pod "bfae2484-2326-45bf-9c9a-fc96c4d9206e" (UID: "bfae2484-2326-45bf-9c9a-fc96c4d9206e"). InnerVolumeSpecName "kube-api-access-rcjsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:55:03.174487 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.174456    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-util" (OuterVolumeSpecName: "util") pod "bfae2484-2326-45bf-9c9a-fc96c4d9206e" (UID: "bfae2484-2326-45bf-9c9a-fc96c4d9206e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:55:03.271713 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.271587    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-bundle\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:55:03.271713 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.271625    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcjsq\" (UniqueName: \"kubernetes.io/projected/bfae2484-2326-45bf-9c9a-fc96c4d9206e-kube-api-access-rcjsq\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:55:03.271713 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.271639    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfae2484-2326-45bf-9c9a-fc96c4d9206e-util\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:55:03.993484 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.993444    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr" event={"ID":"bfae2484-2326-45bf-9c9a-fc96c4d9206e","Type":"ContainerDied","Data":"9900044cf22e9f3fc44b30c4d36adc8bd2a8ec17badf089ac56da04193c62e79"}
Sep 16 12:55:03.993484 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.993469    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rptxr"
Sep 16 12:55:03.993714 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:03.993501    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9900044cf22e9f3fc44b30c4d36adc8bd2a8ec17badf089ac56da04193c62e79"
Sep 16 12:55:07.486991 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:07.486961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:08.246558 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:08.246516    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:11.342145 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:11.342112    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:11.447093 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:11.447064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:13.780699 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:13.780640    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:13.916714 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:13.916677    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:16.080539 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:16.080508    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:16.175945 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:16.175918    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:16.241817 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:16.241791    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:16.786030 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:16.785996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:18.403204 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.403175    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:18.482849 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.482824    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-779f9fc776-fnwtv"]
Sep 16 12:55:18.483120 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.483108    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerName="pull"
Sep 16 12:55:18.483170 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.483125    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerName="pull"
Sep 16 12:55:18.483170 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.483141    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerName="util"
Sep 16 12:55:18.483170 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.483147    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerName="util"
Sep 16 12:55:18.483170 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.483159    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerName="extract"
Sep 16 12:55:18.483170 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.483165    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerName="extract"
Sep 16 12:55:18.483313 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.483217    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfae2484-2326-45bf-9c9a-fc96c4d9206e" containerName="extract"
Sep 16 12:55:18.486919 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.486904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.496817 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.496798    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:55:18.496817 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.496813    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:55:18.496977 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.496813    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:55:18.496977 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.496813    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:55:18.498818 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.498802    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c7h5m\""
Sep 16 12:55:18.498908 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.498893    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:55:18.499385 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.499372    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:55:18.499843 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.499831    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:55:18.504024 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.504009    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:55:18.507701 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.507683    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-779f9fc776-fnwtv"]
Sep 16 12:55:18.545120 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.545090    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:18.579397 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.579377    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-console-config\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.579476 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.579400    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-service-ca\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.579476 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.579427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/597bf9e1-3371-4889-8ef0-8c7c045ee043-console-serving-cert\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.579476 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.579446    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-trusted-ca-bundle\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.579476 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.579467    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-oauth-serving-cert\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.579708 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.579549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/597bf9e1-3371-4889-8ef0-8c7c045ee043-console-oauth-config\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.579708 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.579611    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l5tq\" (UniqueName: \"kubernetes.io/projected/597bf9e1-3371-4889-8ef0-8c7c045ee043-kube-api-access-4l5tq\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.638805 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.638780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:18.680586 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.680537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/597bf9e1-3371-4889-8ef0-8c7c045ee043-console-serving-cert\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.680586 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.680566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-trusted-ca-bundle\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.680727 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.680590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-oauth-serving-cert\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.680727 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.680614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/597bf9e1-3371-4889-8ef0-8c7c045ee043-console-oauth-config\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.680727 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.680671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l5tq\" (UniqueName: \"kubernetes.io/projected/597bf9e1-3371-4889-8ef0-8c7c045ee043-kube-api-access-4l5tq\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.680727 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.680694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-console-config\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.680915 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.680733    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-service-ca\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.681410 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.681386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-service-ca\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.681519 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.681386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-console-config\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.681519 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.681504    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-trusted-ca-bundle\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.681712 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.681692    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/597bf9e1-3371-4889-8ef0-8c7c045ee043-oauth-serving-cert\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.683020 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.682999    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/597bf9e1-3371-4889-8ef0-8c7c045ee043-console-serving-cert\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.683159 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.683144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/597bf9e1-3371-4889-8ef0-8c7c045ee043-console-oauth-config\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.691895 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.691879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l5tq\" (UniqueName: \"kubernetes.io/projected/597bf9e1-3371-4889-8ef0-8c7c045ee043-kube-api-access-4l5tq\") pod \"console-779f9fc776-fnwtv\" (UID: \"597bf9e1-3371-4889-8ef0-8c7c045ee043\") " pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:18.795629 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:18.795605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:19.072684 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.072596    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-779f9fc776-fnwtv"]
Sep 16 12:55:19.075548 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:55:19.075521    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597bf9e1_3371_4889_8ef0_8c7c045ee043.slice/crio-dc7d302472392feed5a06a572d391a25216c35ba9df12aa45eb649116d436688 WatchSource:0}: Error finding container dc7d302472392feed5a06a572d391a25216c35ba9df12aa45eb649116d436688: Status 404 returned error can't find the container with id dc7d302472392feed5a06a572d391a25216c35ba9df12aa45eb649116d436688
Sep 16 12:55:19.605120 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.605088    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"]
Sep 16 12:55:19.608789 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.608772    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.612204 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.612184    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p27d8\""
Sep 16 12:55:19.612348 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.612328    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:55:19.613454 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.613435    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:55:19.622532 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.622504    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"]
Sep 16 12:55:19.688619 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.688595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.688757 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.688626    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nck8q\" (UniqueName: \"kubernetes.io/projected/9feebe76-a82e-40b6-9ce1-020c04db4483-kube-api-access-nck8q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.688829 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.688795    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.789874 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.789844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.789986 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.789885    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.789986 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.789912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nck8q\" (UniqueName: \"kubernetes.io/projected/9feebe76-a82e-40b6-9ce1-020c04db4483-kube-api-access-nck8q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.790225 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.790203    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.790288 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.790235    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.799407 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.799384    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nck8q\" (UniqueName: \"kubernetes.io/projected/9feebe76-a82e-40b6-9ce1-020c04db4483-kube-api-access-nck8q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:19.917863 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:19.917830    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:20.057428 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:20.057399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779f9fc776-fnwtv" event={"ID":"597bf9e1-3371-4889-8ef0-8c7c045ee043","Type":"ContainerStarted","Data":"5171be2195c1ea671bb4c33e5f2dc26ef0eceaf2bbfbcfe2e1bddbddd84deef1"}
Sep 16 12:55:20.057563 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:20.057441    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779f9fc776-fnwtv" event={"ID":"597bf9e1-3371-4889-8ef0-8c7c045ee043","Type":"ContainerStarted","Data":"dc7d302472392feed5a06a572d391a25216c35ba9df12aa45eb649116d436688"}
Sep 16 12:55:20.079783 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:20.079739    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-779f9fc776-fnwtv" podStartSLOduration=2.079720269 podStartE2EDuration="2.079720269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:20.078611352 +0000 UTC m=+441.259754005" watchObservedRunningTime="2026-09-16 12:55:20.079720269 +0000 UTC m=+441.260862925"
Sep 16 12:55:20.192912 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:20.192881    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"]
Sep 16 12:55:20.195259 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:55:20.195236    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9feebe76_a82e_40b6_9ce1_020c04db4483.slice/crio-abd224765d32014674f2ecf202b8fcf93e8d63078b8ea64aca1375ebcbe00fef WatchSource:0}: Error finding container abd224765d32014674f2ecf202b8fcf93e8d63078b8ea64aca1375ebcbe00fef: Status 404 returned error can't find the container with id abd224765d32014674f2ecf202b8fcf93e8d63078b8ea64aca1375ebcbe00fef
Sep 16 12:55:21.064603 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:21.064571    2575 generic.go:356] "Generic (PLEG): container finished" podID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerID="6347e2f2f1affc148ab543c84530fa829ad38a087fba016a5eaacdc37afb0684" exitCode=0
Sep 16 12:55:21.064982 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:21.064681    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss" event={"ID":"9feebe76-a82e-40b6-9ce1-020c04db4483","Type":"ContainerDied","Data":"6347e2f2f1affc148ab543c84530fa829ad38a087fba016a5eaacdc37afb0684"}
Sep 16 12:55:21.064982 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:21.064728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss" event={"ID":"9feebe76-a82e-40b6-9ce1-020c04db4483","Type":"ContainerStarted","Data":"abd224765d32014674f2ecf202b8fcf93e8d63078b8ea64aca1375ebcbe00fef"}
Sep 16 12:55:24.443260 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:24.443226    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:25.084970 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:25.084943    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss" event={"ID":"9feebe76-a82e-40b6-9ce1-020c04db4483","Type":"ContainerStarted","Data":"8083f35e9da3927c3d546033a3f86581312d4c0bc3ab345d68c17a85a4f49cc1"}
Sep 16 12:55:26.090559 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:26.090518    2575 generic.go:356] "Generic (PLEG): container finished" podID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerID="8083f35e9da3927c3d546033a3f86581312d4c0bc3ab345d68c17a85a4f49cc1" exitCode=0
Sep 16 12:55:26.090963 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:26.090605    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss" event={"ID":"9feebe76-a82e-40b6-9ce1-020c04db4483","Type":"ContainerDied","Data":"8083f35e9da3927c3d546033a3f86581312d4c0bc3ab345d68c17a85a4f49cc1"}
Sep 16 12:55:26.384020 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:26.383989    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:26.491234 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:26.491205    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:27.097305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:27.097273    2575 generic.go:356] "Generic (PLEG): container finished" podID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerID="e4153e1578834881fe07c081514fd7a367fe961a45e25f5dcf16026717b62fd7" exitCode=0
Sep 16 12:55:27.097660 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:27.097333    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss" event={"ID":"9feebe76-a82e-40b6-9ce1-020c04db4483","Type":"ContainerDied","Data":"e4153e1578834881fe07c081514fd7a367fe961a45e25f5dcf16026717b62fd7"}
Sep 16 12:55:28.225078 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.225055    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:28.253144 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.253119    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-util\") pod \"9feebe76-a82e-40b6-9ce1-020c04db4483\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") "
Sep 16 12:55:28.253272 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.253180    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nck8q\" (UniqueName: \"kubernetes.io/projected/9feebe76-a82e-40b6-9ce1-020c04db4483-kube-api-access-nck8q\") pod \"9feebe76-a82e-40b6-9ce1-020c04db4483\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") "
Sep 16 12:55:28.253272 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.253214    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-bundle\") pod \"9feebe76-a82e-40b6-9ce1-020c04db4483\" (UID: \"9feebe76-a82e-40b6-9ce1-020c04db4483\") "
Sep 16 12:55:28.253962 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.253788    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-bundle" (OuterVolumeSpecName: "bundle") pod "9feebe76-a82e-40b6-9ce1-020c04db4483" (UID: "9feebe76-a82e-40b6-9ce1-020c04db4483"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:55:28.255953 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.255918    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feebe76-a82e-40b6-9ce1-020c04db4483-kube-api-access-nck8q" (OuterVolumeSpecName: "kube-api-access-nck8q") pod "9feebe76-a82e-40b6-9ce1-020c04db4483" (UID: "9feebe76-a82e-40b6-9ce1-020c04db4483"). InnerVolumeSpecName "kube-api-access-nck8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:55:28.260202 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.260178    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-util" (OuterVolumeSpecName: "util") pod "9feebe76-a82e-40b6-9ce1-020c04db4483" (UID: "9feebe76-a82e-40b6-9ce1-020c04db4483"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:55:28.354156 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.354091    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nck8q\" (UniqueName: \"kubernetes.io/projected/9feebe76-a82e-40b6-9ce1-020c04db4483-kube-api-access-nck8q\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:55:28.354156 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.354123    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-bundle\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:55:28.354156 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.354133    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9feebe76-a82e-40b6-9ce1-020c04db4483-util\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:55:28.796257 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.796226    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:28.796416 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.796312    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:28.800946 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:28.800926    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:29.107761 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:29.107673    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss" event={"ID":"9feebe76-a82e-40b6-9ce1-020c04db4483","Type":"ContainerDied","Data":"abd224765d32014674f2ecf202b8fcf93e8d63078b8ea64aca1375ebcbe00fef"}
Sep 16 12:55:29.107761 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:29.107723    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh7nss"
Sep 16 12:55:29.107761 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:29.107737    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abd224765d32014674f2ecf202b8fcf93e8d63078b8ea64aca1375ebcbe00fef"
Sep 16 12:55:29.111851 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:29.111828    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-779f9fc776-fnwtv"
Sep 16 12:55:29.284121 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:29.284091    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:29.350012 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:29.349984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:29.484095 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:29.484063    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:32.624941 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:32.624907    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:32.720528 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:32.720498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:39.712914 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:39.712880    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:39.784777 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:39.784751    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:41.440824 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:41.440787    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:41.509622 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:41.509588    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:42.358471 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:42.358445    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:47.292517 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.292486    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:47.313030 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.313005    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"]
Sep 16 12:55:47.313324 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.313311    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerName="pull"
Sep 16 12:55:47.313324 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.313325    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerName="pull"
Sep 16 12:55:47.313401 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.313344    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerName="extract"
Sep 16 12:55:47.313401 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.313350    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerName="extract"
Sep 16 12:55:47.313401 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.313356    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerName="util"
Sep 16 12:55:47.313401 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.313361    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerName="util"
Sep 16 12:55:47.313540 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.313407    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9feebe76-a82e-40b6-9ce1-020c04db4483" containerName="extract"
Sep 16 12:55:47.320046 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.320027    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.326569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.326493    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 12:55:47.326569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.326519    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 12:55:47.326569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.326563    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 12:55:47.326928 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.326906    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 12:55:47.327035 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.326963    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:55:47.327103 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.327035    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7lmds\""
Sep 16 12:55:47.327800 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.327786    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:55:47.337717 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.337696    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"]
Sep 16 12:55:47.392155 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.392135    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:47.394173 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.394154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdpr\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-kube-api-access-czdpr\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.394235 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.394183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.394235 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.394202    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.394316 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.394293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.394355 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.394328    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.394355 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.394344    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.394419 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.394370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5215cbb8-4858-4566-82ea-43b6353652ff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.495063 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.495041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.495141 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.495072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.495141 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.495096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.495252 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.495235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5215cbb8-4858-4566-82ea-43b6353652ff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.495310 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.495282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdpr\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-kube-api-access-czdpr\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.495364 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.495318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.495415 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.495349    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.495706 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.495683    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.497550 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.497527    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.497661 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.497547    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5215cbb8-4858-4566-82ea-43b6353652ff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.497742 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.497725    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.497742 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.497736    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.505332 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.505313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.505453 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.505433    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdpr\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-kube-api-access-czdpr\") pod \"istiod-openshift-gateway-7b7995d5cd-dbwvb\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.629482 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.629460    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:47.914590 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:47.914565    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"]
Sep 16 12:55:47.917077 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:55:47.917053    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5215cbb8_4858_4566_82ea_43b6353652ff.slice/crio-d2ba63ec44231c50d869fe42b724f1f19736a9532c75150a67fb34b6c35da1e8 WatchSource:0}: Error finding container d2ba63ec44231c50d869fe42b724f1f19736a9532c75150a67fb34b6c35da1e8: Status 404 returned error can't find the container with id d2ba63ec44231c50d869fe42b724f1f19736a9532c75150a67fb34b6c35da1e8
Sep 16 12:55:48.188241 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:48.188171    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb" event={"ID":"5215cbb8-4858-4566-82ea-43b6353652ff","Type":"ContainerStarted","Data":"d2ba63ec44231c50d869fe42b724f1f19736a9532c75150a67fb34b6c35da1e8"}
Sep 16 12:55:48.543508 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:48.543397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:50.314417 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:50.314381    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 12:55:50.314696 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:50.314451    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 12:55:51.204626 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:51.204597    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb" event={"ID":"5215cbb8-4858-4566-82ea-43b6353652ff","Type":"ContainerStarted","Data":"7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1"}
Sep 16 12:55:51.204829 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:51.204778    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:51.206364 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:51.206343    2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-dbwvb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:55:51.206480 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:51.206392    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb" podUID="5215cbb8-4858-4566-82ea-43b6353652ff" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:55:51.230231 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:51.230184    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb" podStartSLOduration=1.835111678 podStartE2EDuration="4.230163636s" totalImagesPullingTime="2.395051958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:51.227851075 +0000 UTC m=+472.408993752" watchObservedRunningTime="2026-09-16 12:55:51.230163636 +0000 UTC m=+472.411306291"
Sep 16 12:55:51.382092 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:51.382051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:51.478927 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:51.478846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:52.209941 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:52.209915    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:55:53.811804 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:53.811756    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:53.906240 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:53.906189    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:55:56.587995 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:55:56.587967    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:00.085417 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:00.085371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:00.212737 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:00.212697    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:01.395845 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:01.395815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:01.553987 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:01.553954    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:05.177045 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.177018    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"]
Sep 16 12:56:05.180582 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.180566    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.183606 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.183584    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:56:05.183875 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.183860    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p27d8\""
Sep 16 12:56:05.185000 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.184982    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:56:05.192615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.192597    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"]
Sep 16 12:56:05.279064 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.279034    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"]
Sep 16 12:56:05.282286 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.282272    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.293309 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.293282    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"]
Sep 16 12:56:05.331823 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.331798    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.331940 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.331836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7k7\" (UniqueName: \"kubernetes.io/projected/2c3e3c1f-69dc-4393-80ea-9ebd11123646-kube-api-access-sk7k7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.331991 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.331950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.366216 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.366195    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"]
Sep 16 12:56:05.369553 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.369535    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.380105 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.380083    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"]
Sep 16 12:56:05.432522 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.432466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.432522 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.432495    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.432672 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.432531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.432672 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.432550    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.432672 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.432605    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7k7\" (UniqueName: \"kubernetes.io/projected/2c3e3c1f-69dc-4393-80ea-9ebd11123646-kube-api-access-sk7k7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.432780 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.432670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2pd\" (UniqueName: \"kubernetes.io/projected/5184b21a-3516-4580-b14c-c23625f161ca-kube-api-access-4x2pd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.432887 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.432873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.432925 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.432897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.441912 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.441891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7k7\" (UniqueName: \"kubernetes.io/projected/2c3e3c1f-69dc-4393-80ea-9ebd11123646-kube-api-access-sk7k7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.467315 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.467292    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"]
Sep 16 12:56:05.470761 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.470747    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.482811 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.482793    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"]
Sep 16 12:56:05.489161 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.489146    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:05.533863 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.533825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2pd\" (UniqueName: \"kubernetes.io/projected/5184b21a-3516-4580-b14c-c23625f161ca-kube-api-access-4x2pd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.534077 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.534052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.534184 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.534095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw66m\" (UniqueName: \"kubernetes.io/projected/438e639a-6618-4332-8241-ede5d3fb20af-kube-api-access-hw66m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.534184 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.534139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.534184 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.534167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.534856 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.534260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.534856 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.534626    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.534856 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.534780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.543797 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.543776    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2pd\" (UniqueName: \"kubernetes.io/projected/5184b21a-3516-4580-b14c-c23625f161ca-kube-api-access-4x2pd\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.597520 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.597494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:05.635396 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.635337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.635396 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.635383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw66m\" (UniqueName: \"kubernetes.io/projected/438e639a-6618-4332-8241-ede5d3fb20af-kube-api-access-hw66m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.635588 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.635416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.635588 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.635500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.635588 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.635530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.635588 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.635555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkw9\" (UniqueName: \"kubernetes.io/projected/5162214a-0993-493b-acb1-5b52253186b3-kube-api-access-kqkw9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.635888 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.635870    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.636015 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.635994    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.647775 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.647732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw66m\" (UniqueName: \"kubernetes.io/projected/438e639a-6618-4332-8241-ede5d3fb20af-kube-api-access-hw66m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.677932 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.677908    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:05.736185 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.736158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.736304 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.736258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.736304 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.736286    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkw9\" (UniqueName: \"kubernetes.io/projected/5162214a-0993-493b-acb1-5b52253186b3-kube-api-access-kqkw9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.736596 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.736569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.736707 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.736610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.745637 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.745617    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkw9\" (UniqueName: \"kubernetes.io/projected/5162214a-0993-493b-acb1-5b52253186b3-kube-api-access-kqkw9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.760382 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.760361    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"]
Sep 16 12:56:05.762989 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:56:05.762962    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3e3c1f_69dc_4393_80ea_9ebd11123646.slice/crio-cfa66bf92765dc7fcc2563e194c6bfd201d85c175098e7d5dca7c6072079bd27 WatchSource:0}: Error finding container cfa66bf92765dc7fcc2563e194c6bfd201d85c175098e7d5dca7c6072079bd27: Status 404 returned error can't find the container with id cfa66bf92765dc7fcc2563e194c6bfd201d85c175098e7d5dca7c6072079bd27
Sep 16 12:56:05.779490 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.779463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:05.875184 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.875152    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"]
Sep 16 12:56:05.951002 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:56:05.950942    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5184b21a_3516_4580_b14c_c23625f161ca.slice/crio-e405a49ce92e658197d7c0823ea3253afa2ddba9b8657af0f3509bbb40b72121 WatchSource:0}: Error finding container e405a49ce92e658197d7c0823ea3253afa2ddba9b8657af0f3509bbb40b72121: Status 404 returned error can't find the container with id e405a49ce92e658197d7c0823ea3253afa2ddba9b8657af0f3509bbb40b72121
Sep 16 12:56:05.952539 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:05.952510    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"]
Sep 16 12:56:05.954306 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:56:05.954285    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438e639a_6618_4332_8241_ede5d3fb20af.slice/crio-6c8a676019f6f68b082f9802e77c3c3ed31b4d794643f66bed639a60640708f2 WatchSource:0}: Error finding container 6c8a676019f6f68b082f9802e77c3c3ed31b4d794643f66bed639a60640708f2: Status 404 returned error can't find the container with id 6c8a676019f6f68b082f9802e77c3c3ed31b4d794643f66bed639a60640708f2
Sep 16 12:56:06.056620 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.056589    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"]
Sep 16 12:56:06.146432 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:56:06.146401    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5162214a_0993_493b_acb1_5b52253186b3.slice/crio-805efb93c698b9832b61fda2c576da0993a32758fc041a7302d18207e2cd8fb7 WatchSource:0}: Error finding container 805efb93c698b9832b61fda2c576da0993a32758fc041a7302d18207e2cd8fb7: Status 404 returned error can't find the container with id 805efb93c698b9832b61fda2c576da0993a32758fc041a7302d18207e2cd8fb7
Sep 16 12:56:06.267479 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.267446    2575 generic.go:356] "Generic (PLEG): container finished" podID="438e639a-6618-4332-8241-ede5d3fb20af" containerID="27cc2ab7991daaf863d6c6a250e9a9b3b77a274194ce2f5ae61e1159a5c0ccaa" exitCode=0
Sep 16 12:56:06.267934 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.267525    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b" event={"ID":"438e639a-6618-4332-8241-ede5d3fb20af","Type":"ContainerDied","Data":"27cc2ab7991daaf863d6c6a250e9a9b3b77a274194ce2f5ae61e1159a5c0ccaa"}
Sep 16 12:56:06.267934 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.267589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b" event={"ID":"438e639a-6618-4332-8241-ede5d3fb20af","Type":"ContainerStarted","Data":"6c8a676019f6f68b082f9802e77c3c3ed31b4d794643f66bed639a60640708f2"}
Sep 16 12:56:06.269194 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.269113    2575 generic.go:356] "Generic (PLEG): container finished" podID="5184b21a-3516-4580-b14c-c23625f161ca" containerID="b70e1c96d360c9bfa51ff6f77367527d96f5c187869e1fd1d261fc3270291845" exitCode=0
Sep 16 12:56:06.269194 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.269183    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss" event={"ID":"5184b21a-3516-4580-b14c-c23625f161ca","Type":"ContainerDied","Data":"b70e1c96d360c9bfa51ff6f77367527d96f5c187869e1fd1d261fc3270291845"}
Sep 16 12:56:06.269326 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.269220    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss" event={"ID":"5184b21a-3516-4580-b14c-c23625f161ca","Type":"ContainerStarted","Data":"e405a49ce92e658197d7c0823ea3253afa2ddba9b8657af0f3509bbb40b72121"}
Sep 16 12:56:06.271006 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.270982    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerID="746af8e529f90fb63de428234bb0be77ca11b840360a4772cf443917f1361f2c" exitCode=0
Sep 16 12:56:06.271121 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.271072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2" event={"ID":"2c3e3c1f-69dc-4393-80ea-9ebd11123646","Type":"ContainerDied","Data":"746af8e529f90fb63de428234bb0be77ca11b840360a4772cf443917f1361f2c"}
Sep 16 12:56:06.271121 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.271116    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2" event={"ID":"2c3e3c1f-69dc-4393-80ea-9ebd11123646","Type":"ContainerStarted","Data":"cfa66bf92765dc7fcc2563e194c6bfd201d85c175098e7d5dca7c6072079bd27"}
Sep 16 12:56:06.272769 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.272751    2575 generic.go:356] "Generic (PLEG): container finished" podID="5162214a-0993-493b-acb1-5b52253186b3" containerID="02f5cf6a80fdc9b779e82b733ba15e49b9394d92ef47ad644f48c51ec9b4cc96" exitCode=0
Sep 16 12:56:06.272847 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.272818    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w" event={"ID":"5162214a-0993-493b-acb1-5b52253186b3","Type":"ContainerDied","Data":"02f5cf6a80fdc9b779e82b733ba15e49b9394d92ef47ad644f48c51ec9b4cc96"}
Sep 16 12:56:06.272899 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:06.272844    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w" event={"ID":"5162214a-0993-493b-acb1-5b52253186b3","Type":"ContainerStarted","Data":"805efb93c698b9832b61fda2c576da0993a32758fc041a7302d18207e2cd8fb7"}
Sep 16 12:56:07.280161 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:07.280129    2575 generic.go:356] "Generic (PLEG): container finished" podID="438e639a-6618-4332-8241-ede5d3fb20af" containerID="4fb574c6f0941afc3a17da993415f6f529c224a37d03a2555f5fa4357ded664d" exitCode=0
Sep 16 12:56:07.280507 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:07.280177    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b" event={"ID":"438e639a-6618-4332-8241-ede5d3fb20af","Type":"ContainerDied","Data":"4fb574c6f0941afc3a17da993415f6f529c224a37d03a2555f5fa4357ded664d"}
Sep 16 12:56:08.286229 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:08.286190    2575 generic.go:356] "Generic (PLEG): container finished" podID="5162214a-0993-493b-acb1-5b52253186b3" containerID="d2bc329f12ac14882d114cd09b9264f6e38d7c3f38e65db193d8102f3f1fec47" exitCode=0
Sep 16 12:56:08.286615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:08.286281    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w" event={"ID":"5162214a-0993-493b-acb1-5b52253186b3","Type":"ContainerDied","Data":"d2bc329f12ac14882d114cd09b9264f6e38d7c3f38e65db193d8102f3f1fec47"}
Sep 16 12:56:08.288675 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:08.288636    2575 generic.go:356] "Generic (PLEG): container finished" podID="438e639a-6618-4332-8241-ede5d3fb20af" containerID="cc792a5017cf7995a71217f30e9cbd021772ae2e78965b2b0be28b58061ef7ed" exitCode=0
Sep 16 12:56:08.288765 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:08.288682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b" event={"ID":"438e639a-6618-4332-8241-ede5d3fb20af","Type":"ContainerDied","Data":"cc792a5017cf7995a71217f30e9cbd021772ae2e78965b2b0be28b58061ef7ed"}
Sep 16 12:56:08.290469 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:08.290450    2575 generic.go:356] "Generic (PLEG): container finished" podID="5184b21a-3516-4580-b14c-c23625f161ca" containerID="29816fee96b5ae1ee2bcdcde390771390f5fd6a8c4796d63e1fb8f194e566134" exitCode=0
Sep 16 12:56:08.290567 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:08.290517    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss" event={"ID":"5184b21a-3516-4580-b14c-c23625f161ca","Type":"ContainerDied","Data":"29816fee96b5ae1ee2bcdcde390771390f5fd6a8c4796d63e1fb8f194e566134"}
Sep 16 12:56:08.292446 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:08.292423    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerID="22de9ad2280427dfae9ea7f53027a23fb71e86a06e440055f9a84ceab4e2e7f1" exitCode=0
Sep 16 12:56:08.292548 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:08.292483    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2" event={"ID":"2c3e3c1f-69dc-4393-80ea-9ebd11123646","Type":"ContainerDied","Data":"22de9ad2280427dfae9ea7f53027a23fb71e86a06e440055f9a84ceab4e2e7f1"}
Sep 16 12:56:09.299860 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.299829    2575 generic.go:356] "Generic (PLEG): container finished" podID="5162214a-0993-493b-acb1-5b52253186b3" containerID="2f97e21a3fe9d45bfc3ad1c1bcd8edf6c320681e623f736968b30dfa12fc6eb6" exitCode=0
Sep 16 12:56:09.300255 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.299911    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w" event={"ID":"5162214a-0993-493b-acb1-5b52253186b3","Type":"ContainerDied","Data":"2f97e21a3fe9d45bfc3ad1c1bcd8edf6c320681e623f736968b30dfa12fc6eb6"}
Sep 16 12:56:09.302047 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.302022    2575 generic.go:356] "Generic (PLEG): container finished" podID="5184b21a-3516-4580-b14c-c23625f161ca" containerID="4236a9c8e8061b1d94faf2176b9688ec3b07d99529d45ea177940eb2085328c6" exitCode=0
Sep 16 12:56:09.302143 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.302090    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss" event={"ID":"5184b21a-3516-4580-b14c-c23625f161ca","Type":"ContainerDied","Data":"4236a9c8e8061b1d94faf2176b9688ec3b07d99529d45ea177940eb2085328c6"}
Sep 16 12:56:09.304077 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.304054    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerID="bac5ba28d04a757fd6fdc372f1c28274bd672353c5c2f29b5e01d33ef2f3fb53" exitCode=0
Sep 16 12:56:09.304157 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.304138    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2" event={"ID":"2c3e3c1f-69dc-4393-80ea-9ebd11123646","Type":"ContainerDied","Data":"bac5ba28d04a757fd6fdc372f1c28274bd672353c5c2f29b5e01d33ef2f3fb53"}
Sep 16 12:56:09.426656 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.426617    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:09.566359 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.566293    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-bundle\") pod \"438e639a-6618-4332-8241-ede5d3fb20af\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") "
Sep 16 12:56:09.566359 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.566342    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw66m\" (UniqueName: \"kubernetes.io/projected/438e639a-6618-4332-8241-ede5d3fb20af-kube-api-access-hw66m\") pod \"438e639a-6618-4332-8241-ede5d3fb20af\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") "
Sep 16 12:56:09.566520 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.566385    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-util\") pod \"438e639a-6618-4332-8241-ede5d3fb20af\" (UID: \"438e639a-6618-4332-8241-ede5d3fb20af\") "
Sep 16 12:56:09.566781 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.566760    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-bundle" (OuterVolumeSpecName: "bundle") pod "438e639a-6618-4332-8241-ede5d3fb20af" (UID: "438e639a-6618-4332-8241-ede5d3fb20af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:09.568572 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.568548    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438e639a-6618-4332-8241-ede5d3fb20af-kube-api-access-hw66m" (OuterVolumeSpecName: "kube-api-access-hw66m") pod "438e639a-6618-4332-8241-ede5d3fb20af" (UID: "438e639a-6618-4332-8241-ede5d3fb20af"). InnerVolumeSpecName "kube-api-access-hw66m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:09.574073 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.574049    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-util" (OuterVolumeSpecName: "util") pod "438e639a-6618-4332-8241-ede5d3fb20af" (UID: "438e639a-6618-4332-8241-ede5d3fb20af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:09.667170 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.667140    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-bundle\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:09.667170 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.667169    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw66m\" (UniqueName: \"kubernetes.io/projected/438e639a-6618-4332-8241-ede5d3fb20af-kube-api-access-hw66m\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:09.667170 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:09.667181    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/438e639a-6618-4332-8241-ede5d3fb20af-util\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:10.311241 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.311209    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b"
Sep 16 12:56:10.311718 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.311239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nvj6b" event={"ID":"438e639a-6618-4332-8241-ede5d3fb20af","Type":"ContainerDied","Data":"6c8a676019f6f68b082f9802e77c3c3ed31b4d794643f66bed639a60640708f2"}
Sep 16 12:56:10.311718 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.311318    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c8a676019f6f68b082f9802e77c3c3ed31b4d794643f66bed639a60640708f2"
Sep 16 12:56:10.461175 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.461149    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:10.463375 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.463355    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:10.501329 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.501311    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:10.574868 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.574799    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-util\") pod \"5162214a-0993-493b-acb1-5b52253186b3\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") "
Sep 16 12:56:10.574868 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.574840    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqkw9\" (UniqueName: \"kubernetes.io/projected/5162214a-0993-493b-acb1-5b52253186b3-kube-api-access-kqkw9\") pod \"5162214a-0993-493b-acb1-5b52253186b3\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") "
Sep 16 12:56:10.574868 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.574865    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-util\") pod \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") "
Sep 16 12:56:10.575117 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.574904    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-bundle\") pod \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") "
Sep 16 12:56:10.575117 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.574922    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk7k7\" (UniqueName: \"kubernetes.io/projected/2c3e3c1f-69dc-4393-80ea-9ebd11123646-kube-api-access-sk7k7\") pod \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\" (UID: \"2c3e3c1f-69dc-4393-80ea-9ebd11123646\") "
Sep 16 12:56:10.575117 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.574958    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-bundle\") pod \"5162214a-0993-493b-acb1-5b52253186b3\" (UID: \"5162214a-0993-493b-acb1-5b52253186b3\") "
Sep 16 12:56:10.575547 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.575494    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-bundle" (OuterVolumeSpecName: "bundle") pod "2c3e3c1f-69dc-4393-80ea-9ebd11123646" (UID: "2c3e3c1f-69dc-4393-80ea-9ebd11123646"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:10.575547 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.575521    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-bundle" (OuterVolumeSpecName: "bundle") pod "5162214a-0993-493b-acb1-5b52253186b3" (UID: "5162214a-0993-493b-acb1-5b52253186b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:10.577151 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.577130    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5162214a-0993-493b-acb1-5b52253186b3-kube-api-access-kqkw9" (OuterVolumeSpecName: "kube-api-access-kqkw9") pod "5162214a-0993-493b-acb1-5b52253186b3" (UID: "5162214a-0993-493b-acb1-5b52253186b3"). InnerVolumeSpecName "kube-api-access-kqkw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:10.577583 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.577557    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3e3c1f-69dc-4393-80ea-9ebd11123646-kube-api-access-sk7k7" (OuterVolumeSpecName: "kube-api-access-sk7k7") pod "2c3e3c1f-69dc-4393-80ea-9ebd11123646" (UID: "2c3e3c1f-69dc-4393-80ea-9ebd11123646"). InnerVolumeSpecName "kube-api-access-sk7k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:10.580858 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.580839    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-util" (OuterVolumeSpecName: "util") pod "5162214a-0993-493b-acb1-5b52253186b3" (UID: "5162214a-0993-493b-acb1-5b52253186b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:10.580981 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.580962    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-util" (OuterVolumeSpecName: "util") pod "2c3e3c1f-69dc-4393-80ea-9ebd11123646" (UID: "2c3e3c1f-69dc-4393-80ea-9ebd11123646"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:10.675699 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.675672    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-util\") pod \"5184b21a-3516-4580-b14c-c23625f161ca\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") "
Sep 16 12:56:10.675809 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.675728    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-bundle\") pod \"5184b21a-3516-4580-b14c-c23625f161ca\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") "
Sep 16 12:56:10.675809 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.675783    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x2pd\" (UniqueName: \"kubernetes.io/projected/5184b21a-3516-4580-b14c-c23625f161ca-kube-api-access-4x2pd\") pod \"5184b21a-3516-4580-b14c-c23625f161ca\" (UID: \"5184b21a-3516-4580-b14c-c23625f161ca\") "
Sep 16 12:56:10.676030 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.676013    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-util\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:10.676102 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.676036    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqkw9\" (UniqueName: \"kubernetes.io/projected/5162214a-0993-493b-acb1-5b52253186b3-kube-api-access-kqkw9\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:10.676102 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.676050    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-util\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:10.676102 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.676062    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c3e3c1f-69dc-4393-80ea-9ebd11123646-bundle\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:10.676102 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.676075    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7k7\" (UniqueName: \"kubernetes.io/projected/2c3e3c1f-69dc-4393-80ea-9ebd11123646-kube-api-access-sk7k7\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:10.676102 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.676087    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5162214a-0993-493b-acb1-5b52253186b3-bundle\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:10.676286 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.676270    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-bundle" (OuterVolumeSpecName: "bundle") pod "5184b21a-3516-4580-b14c-c23625f161ca" (UID: "5184b21a-3516-4580-b14c-c23625f161ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:10.677910 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.677888    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5184b21a-3516-4580-b14c-c23625f161ca-kube-api-access-4x2pd" (OuterVolumeSpecName: "kube-api-access-4x2pd") pod "5184b21a-3516-4580-b14c-c23625f161ca" (UID: "5184b21a-3516-4580-b14c-c23625f161ca"). InnerVolumeSpecName "kube-api-access-4x2pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:10.681107 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.681091    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-util" (OuterVolumeSpecName: "util") pod "5184b21a-3516-4580-b14c-c23625f161ca" (UID: "5184b21a-3516-4580-b14c-c23625f161ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:56:10.776929 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.776903    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x2pd\" (UniqueName: \"kubernetes.io/projected/5184b21a-3516-4580-b14c-c23625f161ca-kube-api-access-4x2pd\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:10.776929 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.776928    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-util\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:10.777073 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:10.776942    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5184b21a-3516-4580-b14c-c23625f161ca-bundle\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:56:11.317188 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:11.317154    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss" event={"ID":"5184b21a-3516-4580-b14c-c23625f161ca","Type":"ContainerDied","Data":"e405a49ce92e658197d7c0823ea3253afa2ddba9b8657af0f3509bbb40b72121"}
Sep 16 12:56:11.317188 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:11.317183    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad9qss"
Sep 16 12:56:11.317686 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:11.317214    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e405a49ce92e658197d7c0823ea3253afa2ddba9b8657af0f3509bbb40b72121"
Sep 16 12:56:11.319255 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:11.319237    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2"
Sep 16 12:56:11.319384 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:11.319267    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccrvk2" event={"ID":"2c3e3c1f-69dc-4393-80ea-9ebd11123646","Type":"ContainerDied","Data":"cfa66bf92765dc7fcc2563e194c6bfd201d85c175098e7d5dca7c6072079bd27"}
Sep 16 12:56:11.319384 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:11.319306    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa66bf92765dc7fcc2563e194c6bfd201d85c175098e7d5dca7c6072079bd27"
Sep 16 12:56:11.321408 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:11.321393    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w"
Sep 16 12:56:11.321501 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:11.321422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d786p7w" event={"ID":"5162214a-0993-493b-acb1-5b52253186b3","Type":"ContainerDied","Data":"805efb93c698b9832b61fda2c576da0993a32758fc041a7302d18207e2cd8fb7"}
Sep 16 12:56:11.321501 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:11.321460    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805efb93c698b9832b61fda2c576da0993a32758fc041a7302d18207e2cd8fb7"
Sep 16 12:56:14.889728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:14.889696    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:16.975626 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975592    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"]
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975924    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5162214a-0993-493b-acb1-5b52253186b3" containerName="pull"
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975936    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5162214a-0993-493b-acb1-5b52253186b3" containerName="pull"
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975946    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerName="pull"
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975951    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerName="pull"
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975959    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5162214a-0993-493b-acb1-5b52253186b3" containerName="extract"
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975965    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5162214a-0993-493b-acb1-5b52253186b3" containerName="extract"
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975976    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5162214a-0993-493b-acb1-5b52253186b3" containerName="util"
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975981    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5162214a-0993-493b-acb1-5b52253186b3" containerName="util"
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975990    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e639a-6618-4332-8241-ede5d3fb20af" containerName="util"
Sep 16 12:56:16.975998 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.975999    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e639a-6618-4332-8241-ede5d3fb20af" containerName="util"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976007    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5184b21a-3516-4580-b14c-c23625f161ca" containerName="extract"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976014    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5184b21a-3516-4580-b14c-c23625f161ca" containerName="extract"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976020    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e639a-6618-4332-8241-ede5d3fb20af" containerName="extract"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976025    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e639a-6618-4332-8241-ede5d3fb20af" containerName="extract"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976031    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerName="extract"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976036    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerName="extract"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976042    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerName="util"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976047    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerName="util"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976054    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5184b21a-3516-4580-b14c-c23625f161ca" containerName="util"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976058    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5184b21a-3516-4580-b14c-c23625f161ca" containerName="util"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976064    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438e639a-6618-4332-8241-ede5d3fb20af" containerName="pull"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976072    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="438e639a-6618-4332-8241-ede5d3fb20af" containerName="pull"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976082    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5184b21a-3516-4580-b14c-c23625f161ca" containerName="pull"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976088    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5184b21a-3516-4580-b14c-c23625f161ca" containerName="pull"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976132    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="438e639a-6618-4332-8241-ede5d3fb20af" containerName="extract"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976140    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5184b21a-3516-4580-b14c-c23625f161ca" containerName="extract"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976149    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5162214a-0993-493b-acb1-5b52253186b3" containerName="extract"
Sep 16 12:56:16.976305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.976158    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c3e3c1f-69dc-4393-80ea-9ebd11123646" containerName="extract"
Sep 16 12:56:16.980344 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.980328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"
Sep 16 12:56:16.983167 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.983146    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:56:16.983167 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.983159    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gvmc5\""
Sep 16 12:56:16.983317 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.983175    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 12:56:16.983708 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.983692    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:56:16.992569 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:16.992549    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"]
Sep 16 12:56:17.122409 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:17.122381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxc4k\" (UniqueName: \"kubernetes.io/projected/6bd138e6-ee11-4e23-85c1-195efe4c7bbc-kube-api-access-zxc4k\") pod \"dns-operator-controller-manager-65dc8f76f5-vgkbm\" (UID: \"6bd138e6-ee11-4e23-85c1-195efe4c7bbc\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"
Sep 16 12:56:17.223308 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:17.223283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxc4k\" (UniqueName: \"kubernetes.io/projected/6bd138e6-ee11-4e23-85c1-195efe4c7bbc-kube-api-access-zxc4k\") pod \"dns-operator-controller-manager-65dc8f76f5-vgkbm\" (UID: \"6bd138e6-ee11-4e23-85c1-195efe4c7bbc\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"
Sep 16 12:56:17.254082 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:17.254022    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxc4k\" (UniqueName: \"kubernetes.io/projected/6bd138e6-ee11-4e23-85c1-195efe4c7bbc-kube-api-access-zxc4k\") pod \"dns-operator-controller-manager-65dc8f76f5-vgkbm\" (UID: \"6bd138e6-ee11-4e23-85c1-195efe4c7bbc\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"
Sep 16 12:56:17.290676 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:17.290624    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"
Sep 16 12:56:17.479397 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:17.479360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:17.566580 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:17.566560    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"]
Sep 16 12:56:17.568212 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:56:17.568186    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd138e6_ee11_4e23_85c1_195efe4c7bbc.slice/crio-0a16af00d014a8fbb68e077261a8ef84c9a98eeda098b03b36b79702cff75ab9 WatchSource:0}: Error finding container 0a16af00d014a8fbb68e077261a8ef84c9a98eeda098b03b36b79702cff75ab9: Status 404 returned error can't find the container with id 0a16af00d014a8fbb68e077261a8ef84c9a98eeda098b03b36b79702cff75ab9
Sep 16 12:56:17.890276 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:17.890233    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:18.316068 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:18.315980    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:18.357476 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:18.357442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm" event={"ID":"6bd138e6-ee11-4e23-85c1-195efe4c7bbc","Type":"ContainerStarted","Data":"0a16af00d014a8fbb68e077261a8ef84c9a98eeda098b03b36b79702cff75ab9"}
Sep 16 12:56:19.185091 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:19.185064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:20.367926 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:20.367898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm" event={"ID":"6bd138e6-ee11-4e23-85c1-195efe4c7bbc","Type":"ContainerStarted","Data":"cc22df815ec6ae3e71e2229fce2d2a1b4873b185792321702c428b974f0623f1"}
Sep 16 12:56:20.368217 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:20.368029    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"
Sep 16 12:56:20.391555 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:20.391506    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm" podStartSLOduration=1.715797834 podStartE2EDuration="4.391492836s" totalImagesPullingTime="2.675695002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:20.389746518 +0000 UTC m=+501.570889172" watchObservedRunningTime="2026-09-16 12:56:20.391492836 +0000 UTC m=+501.572635490"
Sep 16 12:56:20.977728 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:20.977691    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:21.085767 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.085737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:21.420799 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.420771    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"]
Sep 16 12:56:21.424211 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.424190    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:21.427028 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.427005    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jz7gs\""
Sep 16 12:56:21.439382 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.439360    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"]
Sep 16 12:56:21.483605 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.483571    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:21.558052 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.558023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d80a30b-da72-4bcc-8997-240c5f24ac78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-8stk4\" (UID: \"8d80a30b-da72-4bcc-8997-240c5f24ac78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:21.558159 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.558055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7gj7\" (UniqueName: \"kubernetes.io/projected/8d80a30b-da72-4bcc-8997-240c5f24ac78-kube-api-access-l7gj7\") pod \"kuadrant-operator-controller-manager-6d4cb79c-8stk4\" (UID: \"8d80a30b-da72-4bcc-8997-240c5f24ac78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:21.658838 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.658806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d80a30b-da72-4bcc-8997-240c5f24ac78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-8stk4\" (UID: \"8d80a30b-da72-4bcc-8997-240c5f24ac78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:21.658942 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.658846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7gj7\" (UniqueName: \"kubernetes.io/projected/8d80a30b-da72-4bcc-8997-240c5f24ac78-kube-api-access-l7gj7\") pod \"kuadrant-operator-controller-manager-6d4cb79c-8stk4\" (UID: \"8d80a30b-da72-4bcc-8997-240c5f24ac78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:21.659156 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.659139    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d80a30b-da72-4bcc-8997-240c5f24ac78-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-8stk4\" (UID: \"8d80a30b-da72-4bcc-8997-240c5f24ac78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:21.672292 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.672227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7gj7\" (UniqueName: \"kubernetes.io/projected/8d80a30b-da72-4bcc-8997-240c5f24ac78-kube-api-access-l7gj7\") pod \"kuadrant-operator-controller-manager-6d4cb79c-8stk4\" (UID: \"8d80a30b-da72-4bcc-8997-240c5f24ac78\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:21.736246 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.736223    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:21.895200 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:21.895165    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:22.019980 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:22.019915    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"]
Sep 16 12:56:22.029106 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:56:22.029080    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d80a30b_da72_4bcc_8997_240c5f24ac78.slice/crio-f24e0658065b44e314e6dac59d0b8e503ba0a793badcd21853acae01cd1ade67 WatchSource:0}: Error finding container f24e0658065b44e314e6dac59d0b8e503ba0a793badcd21853acae01cd1ade67: Status 404 returned error can't find the container with id f24e0658065b44e314e6dac59d0b8e503ba0a793badcd21853acae01cd1ade67
Sep 16 12:56:22.334737 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:22.334669    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:22.379301 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:22.379271    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4" event={"ID":"8d80a30b-da72-4bcc-8997-240c5f24ac78","Type":"ContainerStarted","Data":"f24e0658065b44e314e6dac59d0b8e503ba0a793badcd21853acae01cd1ade67"}
Sep 16 12:56:22.459918 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:22.459893    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:24.236747 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.236711    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-5swp5"]
Sep 16 12:56:24.242438 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.242421    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5"
Sep 16 12:56:24.245580 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.245560    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wrr6l\""
Sep 16 12:56:24.259762 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.259743    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-5swp5"]
Sep 16 12:56:24.275437 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.275411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvtq\" (UniqueName: \"kubernetes.io/projected/0713248b-14d2-4a6b-8b5e-4dd3f5612560-kube-api-access-7dvtq\") pod \"authorino-operator-6fc8f864f4-5swp5\" (UID: \"0713248b-14d2-4a6b-8b5e-4dd3f5612560\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5"
Sep 16 12:56:24.376135 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.376107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvtq\" (UniqueName: \"kubernetes.io/projected/0713248b-14d2-4a6b-8b5e-4dd3f5612560-kube-api-access-7dvtq\") pod \"authorino-operator-6fc8f864f4-5swp5\" (UID: \"0713248b-14d2-4a6b-8b5e-4dd3f5612560\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5"
Sep 16 12:56:24.397981 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.397962    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvtq\" (UniqueName: \"kubernetes.io/projected/0713248b-14d2-4a6b-8b5e-4dd3f5612560-kube-api-access-7dvtq\") pod \"authorino-operator-6fc8f864f4-5swp5\" (UID: \"0713248b-14d2-4a6b-8b5e-4dd3f5612560\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5"
Sep 16 12:56:24.552615 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.552524    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5"
Sep 16 12:56:24.693701 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.693671    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:24.828576 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:24.828507    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-5swp5"]
Sep 16 12:56:24.831655 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:56:24.831606    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0713248b_14d2_4a6b_8b5e_4dd3f5612560.slice/crio-80f1ccb255355dffc41014fe8369defad861ae8c76120cd71e7575c211842727 WatchSource:0}: Error finding container 80f1ccb255355dffc41014fe8369defad861ae8c76120cd71e7575c211842727: Status 404 returned error can't find the container with id 80f1ccb255355dffc41014fe8369defad861ae8c76120cd71e7575c211842727
Sep 16 12:56:25.395043 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:25.395004    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5" event={"ID":"0713248b-14d2-4a6b-8b5e-4dd3f5612560","Type":"ContainerStarted","Data":"80f1ccb255355dffc41014fe8369defad861ae8c76120cd71e7575c211842727"}
Sep 16 12:56:26.401110 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:26.401070    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5" event={"ID":"0713248b-14d2-4a6b-8b5e-4dd3f5612560","Type":"ContainerStarted","Data":"eb9ca6dd80c5eb1d88ab797261f3b70ff3803feadd4607d1137f94dad1c30e11"}
Sep 16 12:56:26.401504 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:26.401202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5"
Sep 16 12:56:26.428339 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:26.428287    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5" podStartSLOduration=0.948205035 podStartE2EDuration="2.428272869s" totalImagesPullingTime="1.480067834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:26.425886963 +0000 UTC m=+507.607029617" watchObservedRunningTime="2026-09-16 12:56:26.428272869 +0000 UTC m=+507.609415521"
Sep 16 12:56:26.536885 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:26.536856    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:27.713661 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:27.713598    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:27.813087 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:27.813049    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:31.376077 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:31.376049    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-vgkbm"
Sep 16 12:56:32.365261 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:32.365226    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:32.468823 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:32.468795    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:32.568864 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:32.568834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:33.508568 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:33.508536    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:33.577087 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:33.577058    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:37.409133 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:37.409106    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-5swp5"
Sep 16 12:56:40.470537 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:40.470502    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4" event={"ID":"8d80a30b-da72-4bcc-8997-240c5f24ac78","Type":"ContainerStarted","Data":"bb47abb50658ad59510bd352742573d34b7cde9f45b02b550aa99723e75c321d"}
Sep 16 12:56:40.470952 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:40.470594    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:40.515289 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:40.515203    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4" podStartSLOduration=1.2865347630000001 podStartE2EDuration="19.515187535s" totalImagesPullingTime="18.228652772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:40.511884833 +0000 UTC m=+521.693027511" watchObservedRunningTime="2026-09-16 12:56:40.515187535 +0000 UTC m=+521.696330188"
Sep 16 12:56:41.161709 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:41.161676    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:41.260372 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:41.260346    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:42.571490 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:42.571460    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:43.127542 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:43.127502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:44.129983 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:44.129958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:44.231878 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:44.231854    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:46.802364 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:46.802335    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:46.902359 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:46.902327    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:51.361025 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:51.360952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:51.464593 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:51.464568    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:56:51.477346 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:51.477324    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-8stk4"
Sep 16 12:56:51.562453 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:56:51.562420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:00.351933 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:00.351901    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:00.451398 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:00.451371    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:03.252392 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:03.252360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:03.369666 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:03.369618    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:05.838098 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:05.838062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:05.939154 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:05.939120    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:11.232990 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:11.232939    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:11.343291 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:11.343249    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:11.647518 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:11.647486    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:11.749265 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:11.749219    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:12.051775 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:12.051701    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:12.120049 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:12.120017    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:20.068740 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:20.068710    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:20.167810 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:20.167780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:20.267279 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:20.267255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:22.333427 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:22.333397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:23.057461 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:23.057432    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:23.152309 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:23.152280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:24.795371 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:24.795339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:24.898620 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:24.898588    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:24.998519 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:24.998476    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:25.220434 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.220402    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sd62q"]
Sep 16 12:57:25.222487 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.222471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:25.225779 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.225755    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-54rsc\""
Sep 16 12:57:25.225779 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.225767    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 12:57:25.235910 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.235889    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sd62q"]
Sep 16 12:57:25.249804 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.249785    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sd62q"]
Sep 16 12:57:25.323063 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.323041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zdmn\" (UniqueName: \"kubernetes.io/projected/1717eb9d-18fe-448a-852e-d18d95ef4b97-kube-api-access-5zdmn\") pod \"limitador-limitador-84994f7ffc-sd62q\" (UID: \"1717eb9d-18fe-448a-852e-d18d95ef4b97\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:25.323192 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.323096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1717eb9d-18fe-448a-852e-d18d95ef4b97-config-file\") pod \"limitador-limitador-84994f7ffc-sd62q\" (UID: \"1717eb9d-18fe-448a-852e-d18d95ef4b97\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:25.423530 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.423501    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zdmn\" (UniqueName: \"kubernetes.io/projected/1717eb9d-18fe-448a-852e-d18d95ef4b97-kube-api-access-5zdmn\") pod \"limitador-limitador-84994f7ffc-sd62q\" (UID: \"1717eb9d-18fe-448a-852e-d18d95ef4b97\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:25.423690 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.423546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1717eb9d-18fe-448a-852e-d18d95ef4b97-config-file\") pod \"limitador-limitador-84994f7ffc-sd62q\" (UID: \"1717eb9d-18fe-448a-852e-d18d95ef4b97\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:25.424093 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.424076    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1717eb9d-18fe-448a-852e-d18d95ef4b97-config-file\") pod \"limitador-limitador-84994f7ffc-sd62q\" (UID: \"1717eb9d-18fe-448a-852e-d18d95ef4b97\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:25.453349 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.453319    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zdmn\" (UniqueName: \"kubernetes.io/projected/1717eb9d-18fe-448a-852e-d18d95ef4b97-kube-api-access-5zdmn\") pod \"limitador-limitador-84994f7ffc-sd62q\" (UID: \"1717eb9d-18fe-448a-852e-d18d95ef4b97\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:25.532363 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.532298    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:25.819314 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.819291    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sd62q"]
Sep 16 12:57:25.821128 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:57:25.821103    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1717eb9d_18fe_448a_852e_d18d95ef4b97.slice/crio-8353b1e46ae9695368314d176574ca27d197eac834c398c124f361086b546291 WatchSource:0}: Error finding container 8353b1e46ae9695368314d176574ca27d197eac834c398c124f361086b546291: Status 404 returned error can't find the container with id 8353b1e46ae9695368314d176574ca27d197eac834c398c124f361086b546291
Sep 16 12:57:25.834925 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.834901    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:25.936191 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:25.936150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:26.065296 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:26.065271    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:26.690822 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:26.690791    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q" event={"ID":"1717eb9d-18fe-448a-852e-d18d95ef4b97","Type":"ContainerStarted","Data":"8353b1e46ae9695368314d176574ca27d197eac834c398c124f361086b546291"}
Sep 16 12:57:35.160497 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:35.160459    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:35.271564 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:35.271533    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:39.753076 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:39.753041    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q" event={"ID":"1717eb9d-18fe-448a-852e-d18d95ef4b97","Type":"ContainerStarted","Data":"89a3c6e0f59f161ae7d786e37beef6cb5ff37ab0bdc94fe77890ed841668c67b"}
Sep 16 12:57:39.753435 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:39.753111    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:39.775834 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:39.775786    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q" podStartSLOduration=1.6938388629999999 podStartE2EDuration="14.775772288s" totalImagesPullingTime="13.081933425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:57:39.773317845 +0000 UTC m=+580.954460498" watchObservedRunningTime="2026-09-16 12:57:39.775772288 +0000 UTC m=+580.956914941"
Sep 16 12:57:50.758066 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:50.758038    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sd62q"
Sep 16 12:57:56.972890 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:56.972855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:57:57.236941 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:57:57.236863    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:05.831449 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:05.831414    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:05.926748 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:05.926719    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:10.370535 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:10.370506    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:10.476188 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:10.476160    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:11.567356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:11.567328    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:26.720676 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:26.720572    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"]
Sep 16 12:58:26.721216 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:26.720816    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb" podUID="5215cbb8-4858-4566-82ea-43b6353652ff" containerName="discovery" containerID="cri-o://7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1" gracePeriod=30
Sep 16 12:58:26.961970 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:26.961944    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:58:27.000775 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.000698    2575 generic.go:356] "Generic (PLEG): container finished" podID="5215cbb8-4858-4566-82ea-43b6353652ff" containerID="7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1" exitCode=0
Sep 16 12:58:27.000775 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.000743    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"
Sep 16 12:58:27.000775 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.000767    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb" event={"ID":"5215cbb8-4858-4566-82ea-43b6353652ff","Type":"ContainerDied","Data":"7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1"}
Sep 16 12:58:27.000975 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.000816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb" event={"ID":"5215cbb8-4858-4566-82ea-43b6353652ff","Type":"ContainerDied","Data":"d2ba63ec44231c50d869fe42b724f1f19736a9532c75150a67fb34b6c35da1e8"}
Sep 16 12:58:27.000975 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.000843    2575 scope.go:117] "RemoveContainer" containerID="7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1"
Sep 16 12:58:27.009052 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.009035    2575 scope.go:117] "RemoveContainer" containerID="7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1"
Sep 16 12:58:27.009299 ip-10-0-135-92 kubenswrapper[2575]: E0916 12:58:27.009282    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1\": container with ID starting with 7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1 not found: ID does not exist" containerID="7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1"
Sep 16 12:58:27.009345 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.009306    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1"} err="failed to get container status \"7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1\": rpc error: code = NotFound desc = could not find container \"7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1\": container with ID starting with 7abb04676dca9b2c39401cc1d49203bbb6c873db59c706fff62fbe4c82c8cda1 not found: ID does not exist"
Sep 16 12:58:27.069493 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.069464    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-istio-token\") pod \"5215cbb8-4858-4566-82ea-43b6353652ff\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") "
Sep 16 12:58:27.069688 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.069529    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-kubeconfig\") pod \"5215cbb8-4858-4566-82ea-43b6353652ff\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") "
Sep 16 12:58:27.069688 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.069549    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5215cbb8-4858-4566-82ea-43b6353652ff-local-certs\") pod \"5215cbb8-4858-4566-82ea-43b6353652ff\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") "
Sep 16 12:58:27.069688 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.069568    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czdpr\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-kube-api-access-czdpr\") pod \"5215cbb8-4858-4566-82ea-43b6353652ff\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") "
Sep 16 12:58:27.069688 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.069602    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-dns-cert\") pod \"5215cbb8-4858-4566-82ea-43b6353652ff\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") "
Sep 16 12:58:27.069688 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.069629    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-cacerts\") pod \"5215cbb8-4858-4566-82ea-43b6353652ff\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") "
Sep 16 12:58:27.069688 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.069681    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-ca-configmap\") pod \"5215cbb8-4858-4566-82ea-43b6353652ff\" (UID: \"5215cbb8-4858-4566-82ea-43b6353652ff\") "
Sep 16 12:58:27.070252 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.070200    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5215cbb8-4858-4566-82ea-43b6353652ff" (UID: "5215cbb8-4858-4566-82ea-43b6353652ff"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:58:27.072111 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.072087    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5215cbb8-4858-4566-82ea-43b6353652ff" (UID: "5215cbb8-4858-4566-82ea-43b6353652ff"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:58:27.072111 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.072086    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5215cbb8-4858-4566-82ea-43b6353652ff-local-certs" (OuterVolumeSpecName: "local-certs") pod "5215cbb8-4858-4566-82ea-43b6353652ff" (UID: "5215cbb8-4858-4566-82ea-43b6353652ff"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:58:27.072482 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.072452    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5215cbb8-4858-4566-82ea-43b6353652ff" (UID: "5215cbb8-4858-4566-82ea-43b6353652ff"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:58:27.072553 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.072483    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "5215cbb8-4858-4566-82ea-43b6353652ff" (UID: "5215cbb8-4858-4566-82ea-43b6353652ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:58:27.072553 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.072513    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-cacerts" (OuterVolumeSpecName: "cacerts") pod "5215cbb8-4858-4566-82ea-43b6353652ff" (UID: "5215cbb8-4858-4566-82ea-43b6353652ff"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:58:27.072553 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.072519    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-kube-api-access-czdpr" (OuterVolumeSpecName: "kube-api-access-czdpr") pod "5215cbb8-4858-4566-82ea-43b6353652ff" (UID: "5215cbb8-4858-4566-82ea-43b6353652ff"). InnerVolumeSpecName "kube-api-access-czdpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:58:27.171153 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.171123    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-istio-token\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:58:27.171153 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.171150    2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-kubeconfig\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:58:27.171153 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.171159    2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5215cbb8-4858-4566-82ea-43b6353652ff-local-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:58:27.171352 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.171167    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdpr\" (UniqueName: \"kubernetes.io/projected/5215cbb8-4858-4566-82ea-43b6353652ff-kube-api-access-czdpr\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:58:27.171352 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.171177    2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-dns-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:58:27.171352 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.171186    2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5215cbb8-4858-4566-82ea-43b6353652ff-cacerts\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:58:27.171352 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.171194    2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5215cbb8-4858-4566-82ea-43b6353652ff-istio-csr-ca-configmap\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:58:27.330525 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.330489    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"]
Sep 16 12:58:27.336613 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:27.336591    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-dbwvb"]
Sep 16 12:58:29.327888 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:29.327857    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5215cbb8-4858-4566-82ea-43b6353652ff" path="/var/lib/kubelet/pods/5215cbb8-4858-4566-82ea-43b6353652ff/volumes"
Sep 16 12:58:33.813051 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:33.813002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:33.883573 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:33.883522    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:38.202468 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.202411    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:38.309141 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.309112    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:38.372419 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.372389    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867644d759-csztj"]
Sep 16 12:58:38.372735 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.372722    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5215cbb8-4858-4566-82ea-43b6353652ff" containerName="discovery"
Sep 16 12:58:38.372781 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.372736    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5215cbb8-4858-4566-82ea-43b6353652ff" containerName="discovery"
Sep 16 12:58:38.372819 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.372801    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5215cbb8-4858-4566-82ea-43b6353652ff" containerName="discovery"
Sep 16 12:58:38.375580 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.375565    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:58:38.380182 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.380162    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 12:58:38.380307 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.380162    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qktzb\""
Sep 16 12:58:38.384139 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.384121    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 12:58:38.384259 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.384140    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 12:58:38.398066 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.398039    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-csztj"]
Sep 16 12:58:38.432992 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.432970    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:38.436166 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.436147    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"]
Sep 16 12:58:38.439280 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.439262    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:58:38.442519 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.442494    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-ctnwl\""
Sep 16 12:58:38.442833 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.442815    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 12:58:38.451734 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.451705    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08c0f24-fba8-4db6-9b8e-02906da89451-cert\") pod \"kserve-controller-manager-867644d759-csztj\" (UID: \"b08c0f24-fba8-4db6-9b8e-02906da89451\") " pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:58:38.451830 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.451749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9qt4\" (UniqueName: \"kubernetes.io/projected/b08c0f24-fba8-4db6-9b8e-02906da89451-kube-api-access-h9qt4\") pod \"kserve-controller-manager-867644d759-csztj\" (UID: \"b08c0f24-fba8-4db6-9b8e-02906da89451\") " pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:58:38.477696 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.477619    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"]
Sep 16 12:58:38.479444 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.479426    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-66cld"]
Sep 16 12:58:38.482677 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.482658    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:58:38.486997 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.486980    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jhjnl\""
Sep 16 12:58:38.487460 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.487445    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 12:58:38.496498 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.496478    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-66cld"]
Sep 16 12:58:38.519540 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.519519    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:38.552637 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.552614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prn6r\" (UniqueName: \"kubernetes.io/projected/96ced9ad-3408-4138-9872-7c6d67f718b3-kube-api-access-prn6r\") pod \"llmisvc-controller-manager-6c79ffff65-6qs8d\" (UID: \"96ced9ad-3408-4138-9872-7c6d67f718b3\") " pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:58:38.552736 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.552676    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08c0f24-fba8-4db6-9b8e-02906da89451-cert\") pod \"kserve-controller-manager-867644d759-csztj\" (UID: \"b08c0f24-fba8-4db6-9b8e-02906da89451\") " pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:58:38.552736 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.552692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96ced9ad-3408-4138-9872-7c6d67f718b3-cert\") pod \"llmisvc-controller-manager-6c79ffff65-6qs8d\" (UID: \"96ced9ad-3408-4138-9872-7c6d67f718b3\") " pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:58:38.552736 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.552713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9qt4\" (UniqueName: \"kubernetes.io/projected/b08c0f24-fba8-4db6-9b8e-02906da89451-kube-api-access-h9qt4\") pod \"kserve-controller-manager-867644d759-csztj\" (UID: \"b08c0f24-fba8-4db6-9b8e-02906da89451\") " pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:58:38.552736 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.552731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/158ab7a5-b8de-46f9-8881-aabcd3cbe3e0-data\") pod \"seaweedfs-64568bcd49-66cld\" (UID: \"158ab7a5-b8de-46f9-8881-aabcd3cbe3e0\") " pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:58:38.552895 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.552750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj574\" (UniqueName: \"kubernetes.io/projected/158ab7a5-b8de-46f9-8881-aabcd3cbe3e0-kube-api-access-mj574\") pod \"seaweedfs-64568bcd49-66cld\" (UID: \"158ab7a5-b8de-46f9-8881-aabcd3cbe3e0\") " pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:58:38.555190 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.555167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08c0f24-fba8-4db6-9b8e-02906da89451-cert\") pod \"kserve-controller-manager-867644d759-csztj\" (UID: \"b08c0f24-fba8-4db6-9b8e-02906da89451\") " pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:58:38.564726 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.564206    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9qt4\" (UniqueName: \"kubernetes.io/projected/b08c0f24-fba8-4db6-9b8e-02906da89451-kube-api-access-h9qt4\") pod \"kserve-controller-manager-867644d759-csztj\" (UID: \"b08c0f24-fba8-4db6-9b8e-02906da89451\") " pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:58:38.616982 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.616958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:38.653117 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.653093    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prn6r\" (UniqueName: \"kubernetes.io/projected/96ced9ad-3408-4138-9872-7c6d67f718b3-kube-api-access-prn6r\") pod \"llmisvc-controller-manager-6c79ffff65-6qs8d\" (UID: \"96ced9ad-3408-4138-9872-7c6d67f718b3\") " pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:58:38.653220 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.653154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96ced9ad-3408-4138-9872-7c6d67f718b3-cert\") pod \"llmisvc-controller-manager-6c79ffff65-6qs8d\" (UID: \"96ced9ad-3408-4138-9872-7c6d67f718b3\") " pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:58:38.653220 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.653191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/158ab7a5-b8de-46f9-8881-aabcd3cbe3e0-data\") pod \"seaweedfs-64568bcd49-66cld\" (UID: \"158ab7a5-b8de-46f9-8881-aabcd3cbe3e0\") " pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:58:38.653329 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.653222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj574\" (UniqueName: \"kubernetes.io/projected/158ab7a5-b8de-46f9-8881-aabcd3cbe3e0-kube-api-access-mj574\") pod \"seaweedfs-64568bcd49-66cld\" (UID: \"158ab7a5-b8de-46f9-8881-aabcd3cbe3e0\") " pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:58:38.653570 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.653554    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/158ab7a5-b8de-46f9-8881-aabcd3cbe3e0-data\") pod \"seaweedfs-64568bcd49-66cld\" (UID: \"158ab7a5-b8de-46f9-8881-aabcd3cbe3e0\") " pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:58:38.655560 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.655541    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96ced9ad-3408-4138-9872-7c6d67f718b3-cert\") pod \"llmisvc-controller-manager-6c79ffff65-6qs8d\" (UID: \"96ced9ad-3408-4138-9872-7c6d67f718b3\") " pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:58:38.663434 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.663416    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prn6r\" (UniqueName: \"kubernetes.io/projected/96ced9ad-3408-4138-9872-7c6d67f718b3-kube-api-access-prn6r\") pod \"llmisvc-controller-manager-6c79ffff65-6qs8d\" (UID: \"96ced9ad-3408-4138-9872-7c6d67f718b3\") " pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:58:38.666607 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.666580    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj574\" (UniqueName: \"kubernetes.io/projected/158ab7a5-b8de-46f9-8881-aabcd3cbe3e0-kube-api-access-mj574\") pod \"seaweedfs-64568bcd49-66cld\" (UID: \"158ab7a5-b8de-46f9-8881-aabcd3cbe3e0\") " pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:58:38.685270 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.685249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:58:38.749591 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.749075    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:58:38.794003 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.793953    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:58:38.962413 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.962392    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-csztj"]
Sep 16 12:58:38.963685 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:58:38.963658    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08c0f24_fba8_4db6_9b8e_02906da89451.slice/crio-63aee7cab4ce5b04245a65d33f18363cece44b1ae78b50a9811ccb0ac0b7d8a5 WatchSource:0}: Error finding container 63aee7cab4ce5b04245a65d33f18363cece44b1ae78b50a9811ccb0ac0b7d8a5: Status 404 returned error can't find the container with id 63aee7cab4ce5b04245a65d33f18363cece44b1ae78b50a9811ccb0ac0b7d8a5
Sep 16 12:58:38.964837 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:38.964821    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:58:39.027395 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:39.027374    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"]
Sep 16 12:58:39.028246 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:58:39.028226    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod96ced9ad_3408_4138_9872_7c6d67f718b3.slice/crio-5d616b882bde570cd0b6719c79bf6f26d10f8eea6354a2d586b36ebea7906471 WatchSource:0}: Error finding container 5d616b882bde570cd0b6719c79bf6f26d10f8eea6354a2d586b36ebea7906471: Status 404 returned error can't find the container with id 5d616b882bde570cd0b6719c79bf6f26d10f8eea6354a2d586b36ebea7906471
Sep 16 12:58:39.064533 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:39.064497    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d" event={"ID":"96ced9ad-3408-4138-9872-7c6d67f718b3","Type":"ContainerStarted","Data":"5d616b882bde570cd0b6719c79bf6f26d10f8eea6354a2d586b36ebea7906471"}
Sep 16 12:58:39.065747 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:39.065723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-csztj" event={"ID":"b08c0f24-fba8-4db6-9b8e-02906da89451","Type":"ContainerStarted","Data":"63aee7cab4ce5b04245a65d33f18363cece44b1ae78b50a9811ccb0ac0b7d8a5"}
Sep 16 12:58:39.081818 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:39.081788    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-66cld"]
Sep 16 12:58:39.082495 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:58:39.082470    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158ab7a5_b8de_46f9_8881_aabcd3cbe3e0.slice/crio-a3f4c29b2f450ef776d84ced3ac6354416ba1ba13e3ce531dd3143ff5a3f17ad WatchSource:0}: Error finding container a3f4c29b2f450ef776d84ced3ac6354416ba1ba13e3ce531dd3143ff5a3f17ad: Status 404 returned error can't find the container with id a3f4c29b2f450ef776d84ced3ac6354416ba1ba13e3ce531dd3143ff5a3f17ad
Sep 16 12:58:39.618159 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:39.618112    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:39.717979 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:39.717914    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:39.819566 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:39.819534    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:58:40.077011 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:40.076926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-66cld" event={"ID":"158ab7a5-b8de-46f9-8881-aabcd3cbe3e0","Type":"ContainerStarted","Data":"a3f4c29b2f450ef776d84ced3ac6354416ba1ba13e3ce531dd3143ff5a3f17ad"}
Sep 16 12:58:44.105142 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:44.105110    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d" event={"ID":"96ced9ad-3408-4138-9872-7c6d67f718b3","Type":"ContainerStarted","Data":"a6e8d76578a0b9c219f9dbc5197a08d6c4cf11ffec6d51e275c97d4123023a9d"}
Sep 16 12:58:44.105609 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:44.105214    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:58:44.106704 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:44.106679    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-csztj" event={"ID":"b08c0f24-fba8-4db6-9b8e-02906da89451","Type":"ContainerStarted","Data":"fb0b399402b22092292956b2ab309e5b31be29e4a3b0ee6825245242b295c500"}
Sep 16 12:58:44.106821 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:44.106801    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:58:44.108066 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:44.108047    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-66cld" event={"ID":"158ab7a5-b8de-46f9-8881-aabcd3cbe3e0","Type":"ContainerStarted","Data":"67f7c4fd5b52430bd704228bc5a27b17f5b74c1aed7c47cb0ec033d3a5d2ae8e"}
Sep 16 12:58:44.108213 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:44.108196    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:58:44.130426 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:44.130383    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d" podStartSLOduration=1.70501005 podStartE2EDuration="6.130369897s" totalImagesPullingTime="4.425359847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:44.128037721 +0000 UTC m=+645.309180376" watchObservedRunningTime="2026-09-16 12:58:44.130369897 +0000 UTC m=+645.311512572"
Sep 16 12:58:44.157491 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:44.157450    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867644d759-csztj" podStartSLOduration=1.7393438639999999 podStartE2EDuration="6.157436601s" totalImagesPullingTime="4.418092737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:44.155549539 +0000 UTC m=+645.336692191" watchObservedRunningTime="2026-09-16 12:58:44.157436601 +0000 UTC m=+645.338579290"
Sep 16 12:58:44.175902 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:44.175861    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-66cld" podStartSLOduration=1.674109465 podStartE2EDuration="6.175849331s" totalImagesPullingTime="4.501739866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:44.175252225 +0000 UTC m=+645.356394911" watchObservedRunningTime="2026-09-16 12:58:44.175849331 +0000 UTC m=+645.356992032"
Sep 16 12:58:50.114386 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:58:50.114358    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-66cld"
Sep 16 12:59:15.115123 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:15.115094    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6c79ffff65-6qs8d"
Sep 16 12:59:15.118277 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:15.118253    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:59:24.516420 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:24.516383    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:59:25.093113 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.093080    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867644d759-csztj"]
Sep 16 12:59:25.093296 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.093272    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-867644d759-csztj" podUID="b08c0f24-fba8-4db6-9b8e-02906da89451" containerName="manager" containerID="cri-o://fb0b399402b22092292956b2ab309e5b31be29e4a3b0ee6825245242b295c500" gracePeriod=10
Sep 16 12:59:25.113703 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.113670    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/kserve-controller-manager-867644d759-csztj" podUID="b08c0f24-fba8-4db6-9b8e-02906da89451" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": dial tcp 10.133.0.35:8081: connect: connection refused"
Sep 16 12:59:25.125401 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.125379    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-867644d759-nv6h2"]
Sep 16 12:59:25.127834 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.127820    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:25.139550 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.139531    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-nv6h2"]
Sep 16 12:59:25.211817 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.211792    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwwf\" (UniqueName: \"kubernetes.io/projected/d0b06eed-84d3-4e40-a2cb-2748ad290ce5-kube-api-access-8dwwf\") pod \"kserve-controller-manager-867644d759-nv6h2\" (UID: \"d0b06eed-84d3-4e40-a2cb-2748ad290ce5\") " pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:25.211922 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.211834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0b06eed-84d3-4e40-a2cb-2748ad290ce5-cert\") pod \"kserve-controller-manager-867644d759-nv6h2\" (UID: \"d0b06eed-84d3-4e40-a2cb-2748ad290ce5\") " pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:25.312301 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.312274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwwf\" (UniqueName: \"kubernetes.io/projected/d0b06eed-84d3-4e40-a2cb-2748ad290ce5-kube-api-access-8dwwf\") pod \"kserve-controller-manager-867644d759-nv6h2\" (UID: \"d0b06eed-84d3-4e40-a2cb-2748ad290ce5\") " pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:25.312426 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.312346    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0b06eed-84d3-4e40-a2cb-2748ad290ce5-cert\") pod \"kserve-controller-manager-867644d759-nv6h2\" (UID: \"d0b06eed-84d3-4e40-a2cb-2748ad290ce5\") " pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:25.313057 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.313031    2575 generic.go:356] "Generic (PLEG): container finished" podID="b08c0f24-fba8-4db6-9b8e-02906da89451" containerID="fb0b399402b22092292956b2ab309e5b31be29e4a3b0ee6825245242b295c500" exitCode=0
Sep 16 12:59:25.313151 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.313095    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-csztj" event={"ID":"b08c0f24-fba8-4db6-9b8e-02906da89451","Type":"ContainerDied","Data":"fb0b399402b22092292956b2ab309e5b31be29e4a3b0ee6825245242b295c500"}
Sep 16 12:59:25.314795 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.314772    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0b06eed-84d3-4e40-a2cb-2748ad290ce5-cert\") pod \"kserve-controller-manager-867644d759-nv6h2\" (UID: \"d0b06eed-84d3-4e40-a2cb-2748ad290ce5\") " pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:25.325314 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.325293    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwwf\" (UniqueName: \"kubernetes.io/projected/d0b06eed-84d3-4e40-a2cb-2748ad290ce5-kube-api-access-8dwwf\") pod \"kserve-controller-manager-867644d759-nv6h2\" (UID: \"d0b06eed-84d3-4e40-a2cb-2748ad290ce5\") " pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:25.336496 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.336476    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:59:25.413449 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.413410    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9qt4\" (UniqueName: \"kubernetes.io/projected/b08c0f24-fba8-4db6-9b8e-02906da89451-kube-api-access-h9qt4\") pod \"b08c0f24-fba8-4db6-9b8e-02906da89451\" (UID: \"b08c0f24-fba8-4db6-9b8e-02906da89451\") "
Sep 16 12:59:25.413449 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.413447    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08c0f24-fba8-4db6-9b8e-02906da89451-cert\") pod \"b08c0f24-fba8-4db6-9b8e-02906da89451\" (UID: \"b08c0f24-fba8-4db6-9b8e-02906da89451\") "
Sep 16 12:59:25.415836 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.415800    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08c0f24-fba8-4db6-9b8e-02906da89451-cert" (OuterVolumeSpecName: "cert") pod "b08c0f24-fba8-4db6-9b8e-02906da89451" (UID: "b08c0f24-fba8-4db6-9b8e-02906da89451"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:59:25.415836 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.415809    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08c0f24-fba8-4db6-9b8e-02906da89451-kube-api-access-h9qt4" (OuterVolumeSpecName: "kube-api-access-h9qt4") pod "b08c0f24-fba8-4db6-9b8e-02906da89451" (UID: "b08c0f24-fba8-4db6-9b8e-02906da89451"). InnerVolumeSpecName "kube-api-access-h9qt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:59:25.469497 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.469471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:25.514754 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.514726    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9qt4\" (UniqueName: \"kubernetes.io/projected/b08c0f24-fba8-4db6-9b8e-02906da89451-kube-api-access-h9qt4\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:59:25.514754 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.514753    2575 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08c0f24-fba8-4db6-9b8e-02906da89451-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 12:59:25.747623 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:25.747540    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-867644d759-nv6h2"]
Sep 16 12:59:25.752260 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:59:25.752236    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b06eed_84d3_4e40_a2cb_2748ad290ce5.slice/crio-7f0bf5f1684eb6f88d5d468defbaa8fed375f7962c27f91e65310e23d0f422dc WatchSource:0}: Error finding container 7f0bf5f1684eb6f88d5d468defbaa8fed375f7962c27f91e65310e23d0f422dc: Status 404 returned error can't find the container with id 7f0bf5f1684eb6f88d5d468defbaa8fed375f7962c27f91e65310e23d0f422dc
Sep 16 12:59:26.319069 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:26.319034    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-867644d759-csztj"
Sep 16 12:59:26.319069 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:26.319042    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-csztj" event={"ID":"b08c0f24-fba8-4db6-9b8e-02906da89451","Type":"ContainerDied","Data":"63aee7cab4ce5b04245a65d33f18363cece44b1ae78b50a9811ccb0ac0b7d8a5"}
Sep 16 12:59:26.319305 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:26.319107    2575 scope.go:117] "RemoveContainer" containerID="fb0b399402b22092292956b2ab309e5b31be29e4a3b0ee6825245242b295c500"
Sep 16 12:59:26.321007 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:26.320980    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-nv6h2" event={"ID":"d0b06eed-84d3-4e40-a2cb-2748ad290ce5","Type":"ContainerStarted","Data":"0857c7e81273fc13d60924c9538cedc601fdeb30da0ea0a2ef74557dff97154a"}
Sep 16 12:59:26.321141 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:26.321024    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-867644d759-nv6h2" event={"ID":"d0b06eed-84d3-4e40-a2cb-2748ad290ce5","Type":"ContainerStarted","Data":"7f0bf5f1684eb6f88d5d468defbaa8fed375f7962c27f91e65310e23d0f422dc"}
Sep 16 12:59:26.322242 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:26.321471    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:26.364916 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:26.364864    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-867644d759-nv6h2" podStartSLOduration=0.96877053 podStartE2EDuration="1.364841863s" totalImagesPullingTime="396.071333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:59:26.345716174 +0000 UTC m=+687.526858826" watchObservedRunningTime="2026-09-16 12:59:26.364841863 +0000 UTC m=+687.545984518"
Sep 16 12:59:26.366639 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:26.366617    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-867644d759-csztj"]
Sep 16 12:59:26.370803 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:26.370779    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-867644d759-csztj"]
Sep 16 12:59:27.329258 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:27.329227    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08c0f24-fba8-4db6-9b8e-02906da89451" path="/var/lib/kubelet/pods/b08c0f24-fba8-4db6-9b8e-02906da89451/volumes"
Sep 16 12:59:57.333681 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:57.333591    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-867644d759-nv6h2"
Sep 16 12:59:58.729620 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.729587    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:59:58.814904 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.814876    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-xh77q"]
Sep 16 12:59:58.815230 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.815218    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08c0f24-fba8-4db6-9b8e-02906da89451" containerName="manager"
Sep 16 12:59:58.815274 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.815233    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08c0f24-fba8-4db6-9b8e-02906da89451" containerName="manager"
Sep 16 12:59:58.815308 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.815304    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08c0f24-fba8-4db6-9b8e-02906da89451" containerName="manager"
Sep 16 12:59:58.818177 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.818162    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 12:59:58.822475 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.822454    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 12:59:58.822631 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.822616    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-k6mnm\""
Sep 16 12:59:58.829315 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.829296    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-g4lgs"]
Sep 16 12:59:58.832499 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.832484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:58.837584 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.837561    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 12:59:58.837716 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.837582    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 12:59:58.837837 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.837800    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-s8wlr\""
Sep 16 12:59:58.839041 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.839016    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-xh77q"]
Sep 16 12:59:58.856717 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.856696    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:59:58.857016 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.856990    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-g4lgs"]
Sep 16 12:59:58.956356 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.956331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a431196-638e-4216-ab1a-88befa2c7eb9-cert\") pod \"odh-model-controller-7b966d7984-g4lgs\" (UID: \"9a431196-638e-4216-ab1a-88befa2c7eb9\") " pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:58.956482 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.956362    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plr9r\" (UniqueName: \"kubernetes.io/projected/8d6515d7-2d9b-423e-8ab1-2169a29017a3-kube-api-access-plr9r\") pod \"model-serving-api-7b954ffc95-xh77q\" (UID: \"8d6515d7-2d9b-423e-8ab1-2169a29017a3\") " pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 12:59:58.956482 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.956394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a431196-638e-4216-ab1a-88befa2c7eb9-metrics-certs\") pod \"odh-model-controller-7b966d7984-g4lgs\" (UID: \"9a431196-638e-4216-ab1a-88befa2c7eb9\") " pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:58.956482 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.956451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lr7\" (UniqueName: \"kubernetes.io/projected/9a431196-638e-4216-ab1a-88befa2c7eb9-kube-api-access-c6lr7\") pod \"odh-model-controller-7b966d7984-g4lgs\" (UID: \"9a431196-638e-4216-ab1a-88befa2c7eb9\") " pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:58.956588 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:58.956536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6515d7-2d9b-423e-8ab1-2169a29017a3-tls-certs\") pod \"model-serving-api-7b954ffc95-xh77q\" (UID: \"8d6515d7-2d9b-423e-8ab1-2169a29017a3\") " pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 12:59:59.008639 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.008582    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:59:59.057455 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.057433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a431196-638e-4216-ab1a-88befa2c7eb9-cert\") pod \"odh-model-controller-7b966d7984-g4lgs\" (UID: \"9a431196-638e-4216-ab1a-88befa2c7eb9\") " pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:59.057538 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.057460    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plr9r\" (UniqueName: \"kubernetes.io/projected/8d6515d7-2d9b-423e-8ab1-2169a29017a3-kube-api-access-plr9r\") pod \"model-serving-api-7b954ffc95-xh77q\" (UID: \"8d6515d7-2d9b-423e-8ab1-2169a29017a3\") " pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 12:59:59.057538 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.057481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a431196-638e-4216-ab1a-88befa2c7eb9-metrics-certs\") pod \"odh-model-controller-7b966d7984-g4lgs\" (UID: \"9a431196-638e-4216-ab1a-88befa2c7eb9\") " pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:59.057538 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.057496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lr7\" (UniqueName: \"kubernetes.io/projected/9a431196-638e-4216-ab1a-88befa2c7eb9-kube-api-access-c6lr7\") pod \"odh-model-controller-7b966d7984-g4lgs\" (UID: \"9a431196-638e-4216-ab1a-88befa2c7eb9\") " pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:59.057538 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.057533    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6515d7-2d9b-423e-8ab1-2169a29017a3-tls-certs\") pod \"model-serving-api-7b954ffc95-xh77q\" (UID: \"8d6515d7-2d9b-423e-8ab1-2169a29017a3\") " pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 12:59:59.060104 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.060077    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d6515d7-2d9b-423e-8ab1-2169a29017a3-tls-certs\") pod \"model-serving-api-7b954ffc95-xh77q\" (UID: \"8d6515d7-2d9b-423e-8ab1-2169a29017a3\") " pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 12:59:59.060104 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.060083    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a431196-638e-4216-ab1a-88befa2c7eb9-metrics-certs\") pod \"odh-model-controller-7b966d7984-g4lgs\" (UID: \"9a431196-638e-4216-ab1a-88befa2c7eb9\") " pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:59.060230 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.060132    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a431196-638e-4216-ab1a-88befa2c7eb9-cert\") pod \"odh-model-controller-7b966d7984-g4lgs\" (UID: \"9a431196-638e-4216-ab1a-88befa2c7eb9\") " pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:59.077138 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.077120    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plr9r\" (UniqueName: \"kubernetes.io/projected/8d6515d7-2d9b-423e-8ab1-2169a29017a3-kube-api-access-plr9r\") pod \"model-serving-api-7b954ffc95-xh77q\" (UID: \"8d6515d7-2d9b-423e-8ab1-2169a29017a3\") " pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 12:59:59.079162 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.079144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lr7\" (UniqueName: \"kubernetes.io/projected/9a431196-638e-4216-ab1a-88befa2c7eb9-kube-api-access-c6lr7\") pod \"odh-model-controller-7b966d7984-g4lgs\" (UID: \"9a431196-638e-4216-ab1a-88befa2c7eb9\") " pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:59.128019 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.127998    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 12:59:59.130159 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.130139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 12:59:59.143190 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.143169    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 12:59:59.408326 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.408301    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-xh77q"]
Sep 16 12:59:59.409089 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:59:59.409070    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6515d7_2d9b_423e_8ab1_2169a29017a3.slice/crio-0c0b7fc25c7ad0af6efff8098fa26be812108f2248d690efc8c225b41e2028b1 WatchSource:0}: Error finding container 0c0b7fc25c7ad0af6efff8098fa26be812108f2248d690efc8c225b41e2028b1: Status 404 returned error can't find the container with id 0c0b7fc25c7ad0af6efff8098fa26be812108f2248d690efc8c225b41e2028b1
Sep 16 12:59:59.446354 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.446331    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-g4lgs"]
Sep 16 12:59:59.455396 ip-10-0-135-92 kubenswrapper[2575]: W0916 12:59:59.455369    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a431196_638e_4216_ab1a_88befa2c7eb9.slice/crio-c3d4f6e9e43820407a8540a4b6189d025c1fa4d1e9a6ed3122746ce367e8ebbc WatchSource:0}: Error finding container c3d4f6e9e43820407a8540a4b6189d025c1fa4d1e9a6ed3122746ce367e8ebbc: Status 404 returned error can't find the container with id c3d4f6e9e43820407a8540a4b6189d025c1fa4d1e9a6ed3122746ce367e8ebbc
Sep 16 12:59:59.486494 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.486461    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-g4lgs" event={"ID":"9a431196-638e-4216-ab1a-88befa2c7eb9","Type":"ContainerStarted","Data":"c3d4f6e9e43820407a8540a4b6189d025c1fa4d1e9a6ed3122746ce367e8ebbc"}
Sep 16 12:59:59.487583 ip-10-0-135-92 kubenswrapper[2575]: I0916 12:59:59.487561    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-xh77q" event={"ID":"8d6515d7-2d9b-423e-8ab1-2169a29017a3","Type":"ContainerStarted","Data":"0c0b7fc25c7ad0af6efff8098fa26be812108f2248d690efc8c225b41e2028b1"}
Sep 16 13:00:00.500797 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:00.500746    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:00:00.584028 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:00.583996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:00:03.512961 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:03.512917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-g4lgs" event={"ID":"9a431196-638e-4216-ab1a-88befa2c7eb9","Type":"ContainerStarted","Data":"caa71544ac2bca10be72a654a9b96ba3ca3a33ec176b6203878c3555a53f0401"}
Sep 16 13:00:03.513387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:03.512985    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 13:00:03.514427 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:03.514405    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-xh77q" event={"ID":"8d6515d7-2d9b-423e-8ab1-2169a29017a3","Type":"ContainerStarted","Data":"c02a44f4a8c0b5d14735693be5bc96329b4a5c6f85fd9b0d89842a16b08dcdff"}
Sep 16 13:00:03.514511 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:03.514466    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 13:00:03.540029 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:03.539973    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-g4lgs" podStartSLOduration=1.965962298 podStartE2EDuration="5.539954612s" totalImagesPullingTime="3.573992314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:03.539722917 +0000 UTC m=+724.720865571" watchObservedRunningTime="2026-09-16 13:00:03.539954612 +0000 UTC m=+724.721097270"
Sep 16 13:00:03.567298 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:03.567246    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-xh77q" podStartSLOduration=1.9908389039999999 podStartE2EDuration="5.567230318s" totalImagesPullingTime="3.576391414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:03.565419072 +0000 UTC m=+724.746561726" watchObservedRunningTime="2026-09-16 13:00:03.567230318 +0000 UTC m=+724.748373004"
Sep 16 13:00:14.522120 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:14.522088    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-g4lgs"
Sep 16 13:00:14.524025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:14.524003    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-xh77q"
Sep 16 13:00:15.486714 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.486684    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-86zjv"]
Sep 16 13:00:15.492295 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.492278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-86zjv"
Sep 16 13:00:15.496393 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.496352    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 13:00:15.500706 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.500686    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-86zjv"]
Sep 16 13:00:15.579784 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.579757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96985330-40ab-4d19-b926-4638e10a21af-model-cache\") pod \"s3-init-86zjv\" (UID: \"96985330-40ab-4d19-b926-4638e10a21af\") " pod="kserve/s3-init-86zjv"
Sep 16 13:00:15.579784 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.579786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm55g\" (UniqueName: \"kubernetes.io/projected/96985330-40ab-4d19-b926-4638e10a21af-kube-api-access-dm55g\") pod \"s3-init-86zjv\" (UID: \"96985330-40ab-4d19-b926-4638e10a21af\") " pod="kserve/s3-init-86zjv"
Sep 16 13:00:15.681084 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.681050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96985330-40ab-4d19-b926-4638e10a21af-model-cache\") pod \"s3-init-86zjv\" (UID: \"96985330-40ab-4d19-b926-4638e10a21af\") " pod="kserve/s3-init-86zjv"
Sep 16 13:00:15.681228 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.681092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm55g\" (UniqueName: \"kubernetes.io/projected/96985330-40ab-4d19-b926-4638e10a21af-kube-api-access-dm55g\") pod \"s3-init-86zjv\" (UID: \"96985330-40ab-4d19-b926-4638e10a21af\") " pod="kserve/s3-init-86zjv"
Sep 16 13:00:15.681394 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.681375    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96985330-40ab-4d19-b926-4638e10a21af-model-cache\") pod \"s3-init-86zjv\" (UID: \"96985330-40ab-4d19-b926-4638e10a21af\") " pod="kserve/s3-init-86zjv"
Sep 16 13:00:15.691368 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.691336    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm55g\" (UniqueName: \"kubernetes.io/projected/96985330-40ab-4d19-b926-4638e10a21af-kube-api-access-dm55g\") pod \"s3-init-86zjv\" (UID: \"96985330-40ab-4d19-b926-4638e10a21af\") " pod="kserve/s3-init-86zjv"
Sep 16 13:00:15.801759 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:15.801703    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-86zjv"
Sep 16 13:00:16.081666 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:16.081625    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-86zjv"]
Sep 16 13:00:16.082533 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:00:16.082505    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96985330_40ab_4d19_b926_4638e10a21af.slice/crio-f8ece0efc6d032a818e584b60261dbf74fc2151ad0c8edbe53deb6c5f783151e WatchSource:0}: Error finding container f8ece0efc6d032a818e584b60261dbf74fc2151ad0c8edbe53deb6c5f783151e: Status 404 returned error can't find the container with id f8ece0efc6d032a818e584b60261dbf74fc2151ad0c8edbe53deb6c5f783151e
Sep 16 13:00:16.581253 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:16.581221    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-86zjv" event={"ID":"96985330-40ab-4d19-b926-4638e10a21af","Type":"ContainerStarted","Data":"f8ece0efc6d032a818e584b60261dbf74fc2151ad0c8edbe53deb6c5f783151e"}
Sep 16 13:00:21.610380 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:21.610349    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-86zjv" event={"ID":"96985330-40ab-4d19-b926-4638e10a21af","Type":"ContainerStarted","Data":"eb5893c7d13d42e1d344fbe44e2de08ca183b1590c2a0f9dcab4ec7ca35c26c8"}
Sep 16 13:00:23.624041 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:23.624011    2575 generic.go:356] "Generic (PLEG): container finished" podID="96985330-40ab-4d19-b926-4638e10a21af" containerID="eb5893c7d13d42e1d344fbe44e2de08ca183b1590c2a0f9dcab4ec7ca35c26c8" exitCode=0
Sep 16 13:00:23.624381 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:23.624046    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-86zjv" event={"ID":"96985330-40ab-4d19-b926-4638e10a21af","Type":"ContainerDied","Data":"eb5893c7d13d42e1d344fbe44e2de08ca183b1590c2a0f9dcab4ec7ca35c26c8"}
Sep 16 13:00:28.654320 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:28.654286    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-86zjv" event={"ID":"96985330-40ab-4d19-b926-4638e10a21af","Type":"ContainerStarted","Data":"35d340b9905088e78706197c6c26975d44f7ff60e3206f5bca290f6331bc41d3"}
Sep 16 13:00:28.682230 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:00:28.682176    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-86zjv" podStartSLOduration=2.013771005 podStartE2EDuration="13.682161181s" totalImagesPullingTime="9.758062104s" totalInitContainerRuntime="1.910328072s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:28.68114775 +0000 UTC m=+749.862290404" watchObservedRunningTime="2026-09-16 13:00:28.682161181 +0000 UTC m=+749.863303833"
Sep 16 13:01:33.977353 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:33.977314    2575 generic.go:356] "Generic (PLEG): container finished" podID="96985330-40ab-4d19-b926-4638e10a21af" containerID="35d340b9905088e78706197c6c26975d44f7ff60e3206f5bca290f6331bc41d3" exitCode=0
Sep 16 13:01:33.977353 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:33.977351    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-86zjv" event={"ID":"96985330-40ab-4d19-b926-4638e10a21af","Type":"ContainerDied","Data":"35d340b9905088e78706197c6c26975d44f7ff60e3206f5bca290f6331bc41d3"}
Sep 16 13:01:35.111334 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.111309    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-86zjv"
Sep 16 13:01:35.190138 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.190113    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96985330-40ab-4d19-b926-4638e10a21af-model-cache\") pod \"96985330-40ab-4d19-b926-4638e10a21af\" (UID: \"96985330-40ab-4d19-b926-4638e10a21af\") "
Sep 16 13:01:35.190282 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.190178    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm55g\" (UniqueName: \"kubernetes.io/projected/96985330-40ab-4d19-b926-4638e10a21af-kube-api-access-dm55g\") pod \"96985330-40ab-4d19-b926-4638e10a21af\" (UID: \"96985330-40ab-4d19-b926-4638e10a21af\") "
Sep 16 13:01:35.192262 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.192234    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96985330-40ab-4d19-b926-4638e10a21af-kube-api-access-dm55g" (OuterVolumeSpecName: "kube-api-access-dm55g") pod "96985330-40ab-4d19-b926-4638e10a21af" (UID: "96985330-40ab-4d19-b926-4638e10a21af"). InnerVolumeSpecName "kube-api-access-dm55g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:35.208753 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.208731    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96985330-40ab-4d19-b926-4638e10a21af-model-cache" (OuterVolumeSpecName: "model-cache") pod "96985330-40ab-4d19-b926-4638e10a21af" (UID: "96985330-40ab-4d19-b926-4638e10a21af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:01:35.291492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.291442    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96985330-40ab-4d19-b926-4638e10a21af-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:01:35.291492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.291462    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm55g\" (UniqueName: \"kubernetes.io/projected/96985330-40ab-4d19-b926-4638e10a21af-kube-api-access-dm55g\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:01:35.988801 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.988769    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-86zjv"
Sep 16 13:01:35.988963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.988769    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-86zjv" event={"ID":"96985330-40ab-4d19-b926-4638e10a21af","Type":"ContainerDied","Data":"f8ece0efc6d032a818e584b60261dbf74fc2151ad0c8edbe53deb6c5f783151e"}
Sep 16 13:01:35.988963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:35.988871    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8ece0efc6d032a818e584b60261dbf74fc2151ad0c8edbe53deb6c5f783151e"
Sep 16 13:01:36.508727 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:36.508684    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:01:36.633280 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:36.633244    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:01:50.021284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:50.021249    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:01:50.126465 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:01:50.126423    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:02:03.909801 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.909772    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"]
Sep 16 13:02:03.910199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.910124    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96985330-40ab-4d19-b926-4638e10a21af" containerName="s3-init"
Sep 16 13:02:03.910199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.910135    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="96985330-40ab-4d19-b926-4638e10a21af" containerName="s3-init"
Sep 16 13:02:03.910199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.910145    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96985330-40ab-4d19-b926-4638e10a21af" containerName="download-hf-model"
Sep 16 13:02:03.910199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.910151    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="96985330-40ab-4d19-b926-4638e10a21af" containerName="download-hf-model"
Sep 16 13:02:03.910352 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.910222    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="96985330-40ab-4d19-b926-4638e10a21af" containerName="s3-init"
Sep 16 13:02:03.913169 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.913153    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:03.916169 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.916145    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 13:02:03.916447 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.916432    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-j5jv2\""
Sep 16 13:02:03.916493 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.916434    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 13:02:03.917501 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.917479    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 13:02:03.917635 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.917525    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:02:03.917635 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.917551    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 13:02:03.917847 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.917827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:03.917914 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.917853    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c97bab4-32fa-4aae-b90b-da212c3d187e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:03.917914 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.917898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c97bab4-32fa-4aae-b90b-da212c3d187e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:03.918019 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.917933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:03.918019 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.917962    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:03.918122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.918016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz2v6\" (UniqueName: \"kubernetes.io/projected/5c97bab4-32fa-4aae-b90b-da212c3d187e-kube-api-access-nz2v6\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:03.918122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.918040    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:03.918122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.918112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:03.933753 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.933667    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"]
Sep 16 13:02:03.940095 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:03.940072    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:02:04.018783 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.018753    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.018783 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.018787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c97bab4-32fa-4aae-b90b-da212c3d187e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019021 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.018835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c97bab4-32fa-4aae-b90b-da212c3d187e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019021 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.018862    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019021 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.018899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019021 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.018933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz2v6\" (UniqueName: \"kubernetes.io/projected/5c97bab4-32fa-4aae-b90b-da212c3d187e-kube-api-access-nz2v6\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019021 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.018960    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019021 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.018998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019333 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.019212    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019333 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.019250    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.019708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.019787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.019781    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.020019 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.019889    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c97bab4-32fa-4aae-b90b-da212c3d187e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.021260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.021242    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.021558 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.021542    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c97bab4-32fa-4aae-b90b-da212c3d187e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.030280 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.030259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz2v6\" (UniqueName: \"kubernetes.io/projected/5c97bab4-32fa-4aae-b90b-da212c3d187e-kube-api-access-nz2v6\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.223459 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.223393    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:04.506729 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:04.506702    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"]
Sep 16 13:02:04.507742 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:02:04.507718    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c97bab4_32fa_4aae_b90b_da212c3d187e.slice/crio-ea7d7651bf186ad170d6229dabb43beeda1f4dc4685c4b7882e38fb59c4a85c0 WatchSource:0}: Error finding container ea7d7651bf186ad170d6229dabb43beeda1f4dc4685c4b7882e38fb59c4a85c0: Status 404 returned error can't find the container with id ea7d7651bf186ad170d6229dabb43beeda1f4dc4685c4b7882e38fb59c4a85c0
Sep 16 13:02:05.137164 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:05.137122    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" event={"ID":"5c97bab4-32fa-4aae-b90b-da212c3d187e","Type":"ContainerStarted","Data":"26043fb424aecad3bc361a046a5eae6cb5e01f288e9baf95fc946a0e72a2b135"}
Sep 16 13:02:05.137164 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:05.137171    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" event={"ID":"5c97bab4-32fa-4aae-b90b-da212c3d187e","Type":"ContainerStarted","Data":"ea7d7651bf186ad170d6229dabb43beeda1f4dc4685c4b7882e38fb59c4a85c0"}
Sep 16 13:02:07.151561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:07.151477    2575 generic.go:356] "Generic (PLEG): container finished" podID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerID="26043fb424aecad3bc361a046a5eae6cb5e01f288e9baf95fc946a0e72a2b135" exitCode=0
Sep 16 13:02:07.151954 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:07.151550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" event={"ID":"5c97bab4-32fa-4aae-b90b-da212c3d187e","Type":"ContainerDied","Data":"26043fb424aecad3bc361a046a5eae6cb5e01f288e9baf95fc946a0e72a2b135"}
Sep 16 13:02:31.626787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:31.626747    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:02:31.962568 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:31.962462    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:02:37.360732 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:37.360696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" event={"ID":"5c97bab4-32fa-4aae-b90b-da212c3d187e","Type":"ContainerStarted","Data":"9304abe01b890e285a9e646c9463a1f923e2a620cdb10315b7e97d306c0dd647"}
Sep 16 13:02:37.389978 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:37.389927    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podStartSLOduration=2.336728844 podStartE2EDuration="34.389913445s" totalImagesPullingTime="29.720509327s" totalInitContainerRuntime="2.332675274s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:37.389416415 +0000 UTC m=+878.570559079" watchObservedRunningTime="2026-09-16 13:02:37.389913445 +0000 UTC m=+878.571056098"
Sep 16 13:02:38.224036 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:38.223995    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:44.224165 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:44.224127    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:02:44.225441 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:44.225411    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:02:45.327811 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:45.327773    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:02:54.224900 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:02:54.224804    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:03:04.224015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:03:04.223963    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:03:14.224323 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:03:14.224275    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:03:14.610033 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:03:14.609940    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:03:14.677284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:03:14.677251    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:03:24.224231 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:03:24.224184    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:03:34.223821 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:03:34.223776    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:03:44.224251 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:03:44.224203    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:03:54.223967 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:03:54.223920    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:03:57.139407 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:03:57.139353    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:04.223951 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:04.223904    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:04:04.621873 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:04.621843    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:04.721739 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:04.721703    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:10.937550 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:10.937512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:14.224805 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:14.224766    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": dial tcp 10.133.0.42:8000: connect: connection refused"
Sep 16 13:04:16.560089 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:16.560043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:16.631036 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:16.630989    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:21.022382 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:21.022303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:21.375886 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:21.375850    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:24.233886 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:24.233857    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:04:24.242120 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:24.242095    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:04:37.318723 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:37.318689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:42.395362 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:42.395330    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"]
Sep 16 13:04:42.421844 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:42.421815    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:57.418546 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:57.418507    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main" containerID="cri-o://9304abe01b890e285a9e646c9463a1f923e2a620cdb10315b7e97d306c0dd647" gracePeriod=45
Sep 16 13:04:58.136284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.136254    2575 generic.go:356] "Generic (PLEG): container finished" podID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerID="9304abe01b890e285a9e646c9463a1f923e2a620cdb10315b7e97d306c0dd647" exitCode=0
Sep 16 13:04:58.136431 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.136312    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" event={"ID":"5c97bab4-32fa-4aae-b90b-da212c3d187e","Type":"ContainerDied","Data":"9304abe01b890e285a9e646c9463a1f923e2a620cdb10315b7e97d306c0dd647"}
Sep 16 13:04:58.177797 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.177776    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:04:58.294247 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294172    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-kserve-provision-location\") pod \"5c97bab4-32fa-4aae-b90b-da212c3d187e\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") "
Sep 16 13:04:58.294247 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294247    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c97bab4-32fa-4aae-b90b-da212c3d187e-tls-certs\") pod \"5c97bab4-32fa-4aae-b90b-da212c3d187e\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") "
Sep 16 13:04:58.294471 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294268    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-tmp-dir\") pod \"5c97bab4-32fa-4aae-b90b-da212c3d187e\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") "
Sep 16 13:04:58.294471 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294294    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c97bab4-32fa-4aae-b90b-da212c3d187e-cabundle-cert\") pod \"5c97bab4-32fa-4aae-b90b-da212c3d187e\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") "
Sep 16 13:04:58.294471 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294315    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-home\") pod \"5c97bab4-32fa-4aae-b90b-da212c3d187e\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") "
Sep 16 13:04:58.294471 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294440    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-dshm\") pod \"5c97bab4-32fa-4aae-b90b-da212c3d187e\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") "
Sep 16 13:04:58.294722 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294488    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-model-cache\") pod \"5c97bab4-32fa-4aae-b90b-da212c3d187e\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") "
Sep 16 13:04:58.294722 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294536    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz2v6\" (UniqueName: \"kubernetes.io/projected/5c97bab4-32fa-4aae-b90b-da212c3d187e-kube-api-access-nz2v6\") pod \"5c97bab4-32fa-4aae-b90b-da212c3d187e\" (UID: \"5c97bab4-32fa-4aae-b90b-da212c3d187e\") "
Sep 16 13:04:58.294825 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294723    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c97bab4-32fa-4aae-b90b-da212c3d187e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c97bab4-32fa-4aae-b90b-da212c3d187e" (UID: "5c97bab4-32fa-4aae-b90b-da212c3d187e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:04:58.294933 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294899    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c97bab4-32fa-4aae-b90b-da212c3d187e-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:04:58.295007 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.294958    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c97bab4-32fa-4aae-b90b-da212c3d187e" (UID: "5c97bab4-32fa-4aae-b90b-da212c3d187e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:58.295076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.295053    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-home" (OuterVolumeSpecName: "home") pod "5c97bab4-32fa-4aae-b90b-da212c3d187e" (UID: "5c97bab4-32fa-4aae-b90b-da212c3d187e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:58.297106 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.297081    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-dshm" (OuterVolumeSpecName: "dshm") pod "5c97bab4-32fa-4aae-b90b-da212c3d187e" (UID: "5c97bab4-32fa-4aae-b90b-da212c3d187e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:58.297205 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.297094    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c97bab4-32fa-4aae-b90b-da212c3d187e-kube-api-access-nz2v6" (OuterVolumeSpecName: "kube-api-access-nz2v6") pod "5c97bab4-32fa-4aae-b90b-da212c3d187e" (UID: "5c97bab4-32fa-4aae-b90b-da212c3d187e"). InnerVolumeSpecName "kube-api-access-nz2v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:58.297205 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.297141    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c97bab4-32fa-4aae-b90b-da212c3d187e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c97bab4-32fa-4aae-b90b-da212c3d187e" (UID: "5c97bab4-32fa-4aae-b90b-da212c3d187e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:04:58.306934 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.306901    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c97bab4-32fa-4aae-b90b-da212c3d187e" (UID: "5c97bab4-32fa-4aae-b90b-da212c3d187e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:58.315007 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.314988    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c97bab4-32fa-4aae-b90b-da212c3d187e" (UID: "5c97bab4-32fa-4aae-b90b-da212c3d187e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:58.395622 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.395601    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c97bab4-32fa-4aae-b90b-da212c3d187e-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:04:58.395622 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.395619    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:04:58.395776 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.395627    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:04:58.395776 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.395634    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:04:58.395776 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.395664    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:04:58.395776 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.395677    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz2v6\" (UniqueName: \"kubernetes.io/projected/5c97bab4-32fa-4aae-b90b-da212c3d187e-kube-api-access-nz2v6\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:04:58.395776 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:58.395685    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c97bab4-32fa-4aae-b90b-da212c3d187e-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:04:59.143758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:59.143728    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"
Sep 16 13:04:59.144195 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:59.143766    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw" event={"ID":"5c97bab4-32fa-4aae-b90b-da212c3d187e","Type":"ContainerDied","Data":"ea7d7651bf186ad170d6229dabb43beeda1f4dc4685c4b7882e38fb59c4a85c0"}
Sep 16 13:04:59.144195 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:59.143826    2575 scope.go:117] "RemoveContainer" containerID="9304abe01b890e285a9e646c9463a1f923e2a620cdb10315b7e97d306c0dd647"
Sep 16 13:04:59.153852 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:59.153832    2575 scope.go:117] "RemoveContainer" containerID="26043fb424aecad3bc361a046a5eae6cb5e01f288e9baf95fc946a0e72a2b135"
Sep 16 13:04:59.171637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:59.171614    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"]
Sep 16 13:04:59.177338 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:59.177320    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-77f86ff9d85xnqw"]
Sep 16 13:04:59.329853 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:59.329829    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" path="/var/lib/kubelet/pods/5c97bab4-32fa-4aae-b90b-da212c3d187e/volumes"
Sep 16 13:04:59.778101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:59.778067    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:04:59.878592 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:04:59.878561    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:02.499529 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:02.499498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:02.598208 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:02.598177    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:08.527972 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:08.527937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:12.820147 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:12.820091    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:12.969787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:12.969756    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:13.111747 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.111721    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:13.128334 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.128308    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 13:05:13.128670 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.128658    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="storage-initializer"
Sep 16 13:05:13.128729 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.128672    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="storage-initializer"
Sep 16 13:05:13.128729 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.128694    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main"
Sep 16 13:05:13.128729 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.128699    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main"
Sep 16 13:05:13.128836 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.128753    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c97bab4-32fa-4aae-b90b-da212c3d187e" containerName="main"
Sep 16 13:05:13.131771 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.131755    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.134704 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.134679    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-f7qtj\""
Sep 16 13:05:13.135073 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.135055    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 13:05:13.135195 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.135055    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-58zw5\""
Sep 16 13:05:13.136036 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.136018    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 13:05:13.136036 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.136034    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 13:05:13.136260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.136220    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 13:05:13.136398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.136384    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:05:13.145320 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.145300    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 13:05:13.207929 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.207894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f692da-4756-4e99-8fb4-33dd892fc035-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.208127 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.207932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mngp\" (UniqueName: \"kubernetes.io/projected/09f692da-4756-4e99-8fb4-33dd892fc035-kube-api-access-4mngp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.208127 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.208064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.208127 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.208104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.208127 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.208123    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.208378 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.208189    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f692da-4756-4e99-8fb4-33dd892fc035-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.208378 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.208228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.208378 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.208252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.216794 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.216767    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"]
Sep 16 13:05:13.223342 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.223321    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:13.226382 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.226184    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-nnt4h\""
Sep 16 13:05:13.228262 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.228240    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"]
Sep 16 13:05:13.257913 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.257888    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:13.309181 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309281 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309190    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6611f009-f4b0-4df7-bd52-0862a117340a-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d\" (UID: \"6611f009-f4b0-4df7-bd52-0862a117340a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:13.309281 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309281 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309309    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f692da-4756-4e99-8fb4-33dd892fc035-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99q89\" (UniqueName: \"kubernetes.io/projected/6611f009-f4b0-4df7-bd52-0862a117340a-kube-api-access-99q89\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d\" (UID: \"6611f009-f4b0-4df7-bd52-0862a117340a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:13.309398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309554 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309554 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f692da-4756-4e99-8fb4-33dd892fc035-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309554 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mngp\" (UniqueName: \"kubernetes.io/projected/09f692da-4756-4e99-8fb4-33dd892fc035-kube-api-access-4mngp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309554 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309540    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309606    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309700    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.309925 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.309906    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f692da-4756-4e99-8fb4-33dd892fc035-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.311777 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.311756    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.311870 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.311856    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f692da-4756-4e99-8fb4-33dd892fc035-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.318328 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.318310    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mngp\" (UniqueName: \"kubernetes.io/projected/09f692da-4756-4e99-8fb4-33dd892fc035-kube-api-access-4mngp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.409964 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.409896    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6611f009-f4b0-4df7-bd52-0862a117340a-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d\" (UID: \"6611f009-f4b0-4df7-bd52-0862a117340a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:13.409964 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.409937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99q89\" (UniqueName: \"kubernetes.io/projected/6611f009-f4b0-4df7-bd52-0862a117340a-kube-api-access-99q89\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d\" (UID: \"6611f009-f4b0-4df7-bd52-0862a117340a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:13.412323 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.412304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6611f009-f4b0-4df7-bd52-0862a117340a-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d\" (UID: \"6611f009-f4b0-4df7-bd52-0862a117340a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:13.418365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.418346    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99q89\" (UniqueName: \"kubernetes.io/projected/6611f009-f4b0-4df7-bd52-0862a117340a-kube-api-access-99q89\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d\" (UID: \"6611f009-f4b0-4df7-bd52-0862a117340a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:13.442133 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.442108    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:13.537252 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.537222    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:13.721720 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.721696    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 13:05:13.722840 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:05:13.722812    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f692da_4756_4e99_8fb4_33dd892fc035.slice/crio-974546ca5100d7d0b01f538772d485a13639171f608464bb64a3795e0c415f78 WatchSource:0}: Error finding container 974546ca5100d7d0b01f538772d485a13639171f608464bb64a3795e0c415f78: Status 404 returned error can't find the container with id 974546ca5100d7d0b01f538772d485a13639171f608464bb64a3795e0c415f78
Sep 16 13:05:13.808605 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.808582    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"]
Sep 16 13:05:13.809843 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:05:13.809822    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6611f009_f4b0_4df7_bd52_0862a117340a.slice/crio-2e07898618d4af2caddefbf3a2bb9d6e2b1dcd1f3aaafa131d4c8af7b49fe5fa WatchSource:0}: Error finding container 2e07898618d4af2caddefbf3a2bb9d6e2b1dcd1f3aaafa131d4c8af7b49fe5fa: Status 404 returned error can't find the container with id 2e07898618d4af2caddefbf3a2bb9d6e2b1dcd1f3aaafa131d4c8af7b49fe5fa
Sep 16 13:05:13.815425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:13.815408    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:05:14.236759 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:14.236713    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09f692da-4756-4e99-8fb4-33dd892fc035","Type":"ContainerStarted","Data":"b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f"}
Sep 16 13:05:14.237217 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:14.236775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09f692da-4756-4e99-8fb4-33dd892fc035","Type":"ContainerStarted","Data":"974546ca5100d7d0b01f538772d485a13639171f608464bb64a3795e0c415f78"}
Sep 16 13:05:14.238102 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:14.238080    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" event={"ID":"6611f009-f4b0-4df7-bd52-0862a117340a","Type":"ContainerStarted","Data":"2e07898618d4af2caddefbf3a2bb9d6e2b1dcd1f3aaafa131d4c8af7b49fe5fa"}
Sep 16 13:05:15.049497 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:15.049465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:15.105784 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:15.105728    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:17.262257 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:17.262223    2575 generic.go:356] "Generic (PLEG): container finished" podID="09f692da-4756-4e99-8fb4-33dd892fc035" containerID="b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f" exitCode=0
Sep 16 13:05:17.262765 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:17.262298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09f692da-4756-4e99-8fb4-33dd892fc035","Type":"ContainerDied","Data":"b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f"}
Sep 16 13:05:17.264141 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:17.264123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" event={"ID":"6611f009-f4b0-4df7-bd52-0862a117340a","Type":"ContainerStarted","Data":"039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea"}
Sep 16 13:05:17.264258 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:17.264171    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:17.302182 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:17.302133    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" podStartSLOduration=1.8720222789999998 podStartE2EDuration="4.302111947s" totalImagesPullingTime="2.430089668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:17.301461343 +0000 UTC m=+1038.482603997" watchObservedRunningTime="2026-09-16 13:05:17.302111947 +0000 UTC m=+1038.483254599"
Sep 16 13:05:17.743748 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:17.743711    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:18.272906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:18.272868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09f692da-4756-4e99-8fb4-33dd892fc035","Type":"ContainerStarted","Data":"20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc"}
Sep 16 13:05:18.295314 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:18.295267    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.670032558 podStartE2EDuration="5.29525124s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.625218682s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:18.293907123 +0000 UTC m=+1039.475049775" watchObservedRunningTime="2026-09-16 13:05:18.29525124 +0000 UTC m=+1039.476393892"
Sep 16 13:05:18.443175 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:18.443142    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:20.021555 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:20.021519    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:23.442893 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:23.442856    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:05:23.444361 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:23.444335    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:05:33.442939 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:33.442886    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:05:43.442671 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:43.442615    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:05:48.275338 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:48.275304    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:05:49.802513 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:49.802475    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:49.902513 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:49.902475    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:50.007346 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:50.007316    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:50.105918 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:50.105840    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:53.442926 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:53.442829    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:05:53.689081 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:53.689047    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:56.044947 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:56.044908    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:05:57.611404 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:05:57.611365    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:01.184596 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.184565    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"]
Sep 16 13:06:01.188325 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.188303    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.191273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.191249    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 13:06:01.191398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.191252    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 13:06:01.192636 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.192614    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 16 13:06:01.192757 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.192699    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-sp2tl\""
Sep 16 13:06:01.199581 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.199558    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"]
Sep 16 13:06:01.206881 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.206857    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:01.222634 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.222605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec114f78-0ed0-4140-bfd4-049c830206e6-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.222795 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.222678    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.222795 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.222704    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.222795 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.222758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rhf\" (UniqueName: \"kubernetes.io/projected/ec114f78-0ed0-4140-bfd4-049c830206e6-kube-api-access-74rhf\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.222968 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.222805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.222968 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.222876    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.324309 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.324263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.324504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.324356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec114f78-0ed0-4140-bfd4-049c830206e6-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.324504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.324401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.324504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.324425    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.324504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.324451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rhf\" (UniqueName: \"kubernetes.io/projected/ec114f78-0ed0-4140-bfd4-049c830206e6-kube-api-access-74rhf\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.324504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.324483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.324787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.324749    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.324843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.324800    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.325063 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.325035    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.327462 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.327434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.327682 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.327631    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec114f78-0ed0-4140-bfd4-049c830206e6-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.332912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.332889    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rhf\" (UniqueName: \"kubernetes.io/projected/ec114f78-0ed0-4140-bfd4-049c830206e6-kube-api-access-74rhf\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.501945 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.501863    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:01.784966 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:01.784938    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"]
Sep 16 13:06:01.786845 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:06:01.786814    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec114f78_0ed0_4140_bfd4_049c830206e6.slice/crio-1b8e3ae9cdadcc7528a7579df0523adeced2d78697faec0c51afeb3f7437a418 WatchSource:0}: Error finding container 1b8e3ae9cdadcc7528a7579df0523adeced2d78697faec0c51afeb3f7437a418: Status 404 returned error can't find the container with id 1b8e3ae9cdadcc7528a7579df0523adeced2d78697faec0c51afeb3f7437a418
Sep 16 13:06:02.336188 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:02.336153    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:02.456513 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:02.456450    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:02.518220 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:02.518179    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt" event={"ID":"ec114f78-0ed0-4140-bfd4-049c830206e6","Type":"ContainerStarted","Data":"1b8e3ae9cdadcc7528a7579df0523adeced2d78697faec0c51afeb3f7437a418"}
Sep 16 13:06:03.443174 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:03.443145    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:06:03.525872 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:03.525836    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt" event={"ID":"ec114f78-0ed0-4140-bfd4-049c830206e6","Type":"ContainerStarted","Data":"c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889"}
Sep 16 13:06:03.548433 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:03.548376    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt" podStartSLOduration=0.970963407 podStartE2EDuration="2.548357056s" totalImagesPullingTime="1.577393649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:06:03.545865387 +0000 UTC m=+1084.727008041" watchObservedRunningTime="2026-09-16 13:06:03.548357056 +0000 UTC m=+1084.729499711"
Sep 16 13:06:03.704093 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:03.704006    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:03.845343 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:03.845311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:03.943295 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:03.943268    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:04.098325 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:04.098282    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:04.200319 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:04.200282    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:04.502124 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:04.502085    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:07.143288 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.143249    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:07.468398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.468325    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"]
Sep 16 13:06:07.478262 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:06:07.478225    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:06:07.478262 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:06:07.478262 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:06:07.478262 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889" cmd=["/bin/sleep","15"]
Sep 16 13:06:07.478592 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.478280    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt" message="" err=<
Sep 16 13:06:07.478592 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:06:07.478592 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:06:07.478592 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:06:07.478592 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:06:07.478293    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:06:07.478592 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:06:07.478592 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:06:07.478592 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt" podUID="ec114f78-0ed0-4140-bfd4-049c830206e6" containerName="main" containerID="cri-o://c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889"
Sep 16 13:06:07.478592 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.478344    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt" podUID="ec114f78-0ed0-4140-bfd4-049c830206e6" containerName="main" containerID="cri-o://c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889" gracePeriod=60
Sep 16 13:06:07.489214 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.489191    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:07.725095 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.725038    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:07.784706 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.784669    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-model-cache\") pod \"ec114f78-0ed0-4140-bfd4-049c830206e6\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") "
Sep 16 13:06:07.784880 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.784725    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74rhf\" (UniqueName: \"kubernetes.io/projected/ec114f78-0ed0-4140-bfd4-049c830206e6-kube-api-access-74rhf\") pod \"ec114f78-0ed0-4140-bfd4-049c830206e6\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") "
Sep 16 13:06:07.784880 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.784750    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-tmp-dir\") pod \"ec114f78-0ed0-4140-bfd4-049c830206e6\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") "
Sep 16 13:06:07.784880 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.784851    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-dshm\") pod \"ec114f78-0ed0-4140-bfd4-049c830206e6\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") "
Sep 16 13:06:07.785076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.784893    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-home\") pod \"ec114f78-0ed0-4140-bfd4-049c830206e6\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") "
Sep 16 13:06:07.785076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.784925    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec114f78-0ed0-4140-bfd4-049c830206e6" (UID: "ec114f78-0ed0-4140-bfd4-049c830206e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:07.785076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.784962    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec114f78-0ed0-4140-bfd4-049c830206e6" (UID: "ec114f78-0ed0-4140-bfd4-049c830206e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:07.785076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.784973    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec114f78-0ed0-4140-bfd4-049c830206e6-tls-certs\") pod \"ec114f78-0ed0-4140-bfd4-049c830206e6\" (UID: \"ec114f78-0ed0-4140-bfd4-049c830206e6\") "
Sep 16 13:06:07.785299 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.785140    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-home" (OuterVolumeSpecName: "home") pod "ec114f78-0ed0-4140-bfd4-049c830206e6" (UID: "ec114f78-0ed0-4140-bfd4-049c830206e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:07.785299 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.785234    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:07.785299 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.785252    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:07.785299 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.785264    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:07.787091 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.787067    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec114f78-0ed0-4140-bfd4-049c830206e6-kube-api-access-74rhf" (OuterVolumeSpecName: "kube-api-access-74rhf") pod "ec114f78-0ed0-4140-bfd4-049c830206e6" (UID: "ec114f78-0ed0-4140-bfd4-049c830206e6"). InnerVolumeSpecName "kube-api-access-74rhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:06:07.787226 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.787067    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-dshm" (OuterVolumeSpecName: "dshm") pod "ec114f78-0ed0-4140-bfd4-049c830206e6" (UID: "ec114f78-0ed0-4140-bfd4-049c830206e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:07.787226 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.787139    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec114f78-0ed0-4140-bfd4-049c830206e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec114f78-0ed0-4140-bfd4-049c830206e6" (UID: "ec114f78-0ed0-4140-bfd4-049c830206e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:06:07.885866 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.885837    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec114f78-0ed0-4140-bfd4-049c830206e6-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:07.885866 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.885862    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74rhf\" (UniqueName: \"kubernetes.io/projected/ec114f78-0ed0-4140-bfd4-049c830206e6-kube-api-access-74rhf\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:07.885866 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:07.885874    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec114f78-0ed0-4140-bfd4-049c830206e6-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:08.557356 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.557325    2575 generic.go:356] "Generic (PLEG): container finished" podID="ec114f78-0ed0-4140-bfd4-049c830206e6" containerID="c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889" exitCode=0
Sep 16 13:06:08.557783 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.557382    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"
Sep 16 13:06:08.557783 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.557383    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt" event={"ID":"ec114f78-0ed0-4140-bfd4-049c830206e6","Type":"ContainerDied","Data":"c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889"}
Sep 16 13:06:08.557783 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.557509    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt" event={"ID":"ec114f78-0ed0-4140-bfd4-049c830206e6","Type":"ContainerDied","Data":"1b8e3ae9cdadcc7528a7579df0523adeced2d78697faec0c51afeb3f7437a418"}
Sep 16 13:06:08.557783 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.557545    2575 scope.go:117] "RemoveContainer" containerID="c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889"
Sep 16 13:06:08.567135 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.567106    2575 scope.go:117] "RemoveContainer" containerID="c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889"
Sep 16 13:06:08.567444 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:06:08.567423    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889\": container with ID starting with c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889 not found: ID does not exist" containerID="c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889"
Sep 16 13:06:08.567529 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.567453    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889"} err="failed to get container status \"c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889\": rpc error: code = NotFound desc = could not find container \"c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889\": container with ID starting with c573ae5509adf4bf9a52d27d667940ac8da6fffd54c771829f7efb5b7ac9a889 not found: ID does not exist"
Sep 16 13:06:08.582780 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.582752    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"]
Sep 16 13:06:08.587050 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.587031    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-55jnt"]
Sep 16 13:06:08.787987 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:08.787957    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:09.330498 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:09.330465    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec114f78-0ed0-4140-bfd4-049c830206e6" path="/var/lib/kubelet/pods/ec114f78-0ed0-4140-bfd4-049c830206e6/volumes"
Sep 16 13:06:13.443470 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:13.443431    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:06:13.831692 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:13.831581    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:14.860537 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:14.860504    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:19.989691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:19.989625    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:20.232757 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:20.232714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:23.443039 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:23.442984    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:06:24.982347 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.982311    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"]
Sep 16 13:06:24.982987 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.982964    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec114f78-0ed0-4140-bfd4-049c830206e6" containerName="main"
Sep 16 13:06:24.983067 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.982992    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec114f78-0ed0-4140-bfd4-049c830206e6" containerName="main"
Sep 16 13:06:24.983121 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.983106    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec114f78-0ed0-4140-bfd4-049c830206e6" containerName="main"
Sep 16 13:06:24.987679 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.987660    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:24.990307 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.990285    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 13:06:24.991588 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.991554    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-nhhmf\""
Sep 16 13:06:24.991723 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.991595    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 13:06:24.991723 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.991714    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 16 13:06:24.997625 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:24.997602    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"]
Sep 16 13:06:25.024721 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.024674    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:25.132475 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.132442    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.132475 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.132485    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.132789 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.132510    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.132789 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.132608    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.132789 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.132638    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78mr\" (UniqueName: \"kubernetes.io/projected/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-kube-api-access-k78mr\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.132789 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.132698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.233786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.233691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.233786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.233735    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.233786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.233763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.234072 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.233795    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.234072 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.233811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78mr\" (UniqueName: \"kubernetes.io/projected/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-kube-api-access-k78mr\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.234072 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.233838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.234242 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.234158    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.234242 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.234194    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.234242 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.234217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.236296 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.236272    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.236558 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.236537    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.245426 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.245396    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78mr\" (UniqueName: \"kubernetes.io/projected/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-kube-api-access-k78mr\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.299402 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.299375    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:25.585179 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.585151    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"]
Sep 16 13:06:25.586201 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:06:25.586171    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bec9bd_3135_46dc_8174_b8f98ab4f8a5.slice/crio-a7f5377bc3f23f879b9bab85fda6d4b1da4bd245ca518852ec07b9bb9b1244e2 WatchSource:0}: Error finding container a7f5377bc3f23f879b9bab85fda6d4b1da4bd245ca518852ec07b9bb9b1244e2: Status 404 returned error can't find the container with id a7f5377bc3f23f879b9bab85fda6d4b1da4bd245ca518852ec07b9bb9b1244e2
Sep 16 13:06:25.659091 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.659054    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv" event={"ID":"90bec9bd-3135-46dc-8174-b8f98ab4f8a5","Type":"ContainerStarted","Data":"e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa"}
Sep 16 13:06:25.659237 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.659149    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv" event={"ID":"90bec9bd-3135-46dc-8174-b8f98ab4f8a5","Type":"ContainerStarted","Data":"a7f5377bc3f23f879b9bab85fda6d4b1da4bd245ca518852ec07b9bb9b1244e2"}
Sep 16 13:06:25.677817 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:25.677753    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv" podStartSLOduration=1.6777320329999998 podStartE2EDuration="1.677732033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:06:25.676923685 +0000 UTC m=+1106.858066340" watchObservedRunningTime="2026-09-16 13:06:25.677732033 +0000 UTC m=+1106.858874687"
Sep 16 13:06:26.299698 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:26.299659    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:27.214079 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:27.214040    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:27.315195 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:27.315158    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:27.421076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:27.421027    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:28.786512 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:28.786483    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"]
Sep 16 13:06:28.796334 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:06:28.796297    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:06:28.796334 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:06:28.796334 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:06:28.796334 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa" cmd=["/bin/sleep","15"]
Sep 16 13:06:28.796669 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:28.796351    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv" message="" err=<
Sep 16 13:06:28.796669 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:06:28.796669 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:06:28.796669 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:06:28.796669 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:06:28.796362    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:06:28.796669 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:06:28.796669 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:06:28.796669 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv" podUID="90bec9bd-3135-46dc-8174-b8f98ab4f8a5" containerName="main" containerID="cri-o://e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa"
Sep 16 13:06:28.796669 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:28.796408    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv" podUID="90bec9bd-3135-46dc-8174-b8f98ab4f8a5" containerName="main" containerID="cri-o://e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa" gracePeriod=60
Sep 16 13:06:28.809100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:28.809076    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:29.048886 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.048834    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:29.169592 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.169564    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-home\") pod \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") "
Sep 16 13:06:29.169814 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.169602    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-model-cache\") pod \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") "
Sep 16 13:06:29.169814 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.169669    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tmp-dir\") pod \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") "
Sep 16 13:06:29.169814 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.169734    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-dshm\") pod \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") "
Sep 16 13:06:29.169814 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.169772    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tls-certs\") pod \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") "
Sep 16 13:06:29.169814 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.169803    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k78mr\" (UniqueName: \"kubernetes.io/projected/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-kube-api-access-k78mr\") pod \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\" (UID: \"90bec9bd-3135-46dc-8174-b8f98ab4f8a5\") "
Sep 16 13:06:29.170101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.169855    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-home" (OuterVolumeSpecName: "home") pod "90bec9bd-3135-46dc-8174-b8f98ab4f8a5" (UID: "90bec9bd-3135-46dc-8174-b8f98ab4f8a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:29.170101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.169883    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "90bec9bd-3135-46dc-8174-b8f98ab4f8a5" (UID: "90bec9bd-3135-46dc-8174-b8f98ab4f8a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:29.170101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.170008    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90bec9bd-3135-46dc-8174-b8f98ab4f8a5" (UID: "90bec9bd-3135-46dc-8174-b8f98ab4f8a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:29.170236 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.170102    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:29.170236 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.170119    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:29.170236 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.170133    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:29.172719 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.172636    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-dshm" (OuterVolumeSpecName: "dshm") pod "90bec9bd-3135-46dc-8174-b8f98ab4f8a5" (UID: "90bec9bd-3135-46dc-8174-b8f98ab4f8a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:06:29.172846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.172761    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90bec9bd-3135-46dc-8174-b8f98ab4f8a5" (UID: "90bec9bd-3135-46dc-8174-b8f98ab4f8a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:06:29.172846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.172769    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-kube-api-access-k78mr" (OuterVolumeSpecName: "kube-api-access-k78mr") pod "90bec9bd-3135-46dc-8174-b8f98ab4f8a5" (UID: "90bec9bd-3135-46dc-8174-b8f98ab4f8a5"). InnerVolumeSpecName "kube-api-access-k78mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:06:29.270726 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.270691    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k78mr\" (UniqueName: \"kubernetes.io/projected/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-kube-api-access-k78mr\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:29.270726 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.270723    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:29.270726 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.270733    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90bec9bd-3135-46dc-8174-b8f98ab4f8a5-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:06:29.683964 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.683925    2575 generic.go:356] "Generic (PLEG): container finished" podID="90bec9bd-3135-46dc-8174-b8f98ab4f8a5" containerID="e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa" exitCode=0
Sep 16 13:06:29.684122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.683969    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"
Sep 16 13:06:29.684122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.683994    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv" event={"ID":"90bec9bd-3135-46dc-8174-b8f98ab4f8a5","Type":"ContainerDied","Data":"e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa"}
Sep 16 13:06:29.684122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.684054    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv" event={"ID":"90bec9bd-3135-46dc-8174-b8f98ab4f8a5","Type":"ContainerDied","Data":"a7f5377bc3f23f879b9bab85fda6d4b1da4bd245ca518852ec07b9bb9b1244e2"}
Sep 16 13:06:29.684122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.684084    2575 scope.go:117] "RemoveContainer" containerID="e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa"
Sep 16 13:06:29.693584 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.693566    2575 scope.go:117] "RemoveContainer" containerID="e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa"
Sep 16 13:06:29.693888 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:06:29.693862    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa\": container with ID starting with e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa not found: ID does not exist" containerID="e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa"
Sep 16 13:06:29.694010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.693894    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa"} err="failed to get container status \"e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa\": rpc error: code = NotFound desc = could not find container \"e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa\": container with ID starting with e9418951c17d3072388894c6e283e50f6e617914e540d9e86f165592009579fa not found: ID does not exist"
Sep 16 13:06:29.701945 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.701914    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"]
Sep 16 13:06:29.708930 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:29.708873    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf95vqv"]
Sep 16 13:06:30.169101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:30.169068    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:30.270367 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:30.270332    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:31.328950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:31.328911    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90bec9bd-3135-46dc-8174-b8f98ab4f8a5" path="/var/lib/kubelet/pods/90bec9bd-3135-46dc-8174-b8f98ab4f8a5/volumes"
Sep 16 13:06:32.277758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:32.277726    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:33.443050 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:33.443001    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:06:37.775939 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:37.775902    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:38.328768 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:38.328731    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:43.442824 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:43.442779    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:06:44.289759 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:44.289721    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:46.076075 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.076040    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"]
Sep 16 13:06:46.076461 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.076447    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90bec9bd-3135-46dc-8174-b8f98ab4f8a5" containerName="main"
Sep 16 13:06:46.076505 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.076463    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="90bec9bd-3135-46dc-8174-b8f98ab4f8a5" containerName="main"
Sep 16 13:06:46.076544 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.076531    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="90bec9bd-3135-46dc-8174-b8f98ab4f8a5" containerName="main"
Sep 16 13:06:46.084869 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.084849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.088313 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.088290    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-jljl5\""
Sep 16 13:06:46.088758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.088456    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 13:06:46.088758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.088319    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 13:06:46.088758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.088338    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 13:06:46.088758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.088420    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:06:46.088758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.088290    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 13:06:46.089776 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.089758    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"]
Sep 16 13:06:46.104902 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.104874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kserve-provision-location\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.105035 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.104932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-model-cache\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.105035 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.104959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tmp-dir\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.105141 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.105069    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmh96\" (UniqueName: \"kubernetes.io/projected/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kube-api-access-zmh96\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.105260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.105146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-home\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.105260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.105175    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-cabundle-cert\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.105260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.105210    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-dshm\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.105260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.105255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tls-certs\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.186127 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.186095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:46.206390 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.206357    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmh96\" (UniqueName: \"kubernetes.io/projected/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kube-api-access-zmh96\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.206561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.206409    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-home\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.206561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.206435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-cabundle-cert\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.206561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.206461    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-dshm\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.206561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.206485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tls-certs\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.206561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.206523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kserve-provision-location\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.206561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.206557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-model-cache\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.206886 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.206587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tmp-dir\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.206990 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.206962    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-home\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.207164 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.207019    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-model-cache\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.207475 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.207455    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kserve-provision-location\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.207541 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.207460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tmp-dir\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.207605 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.207583    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-cabundle-cert\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.208912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.208893    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-dshm\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.209160 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.209142    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tls-certs\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.214377 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.214357    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmh96\" (UniqueName: \"kubernetes.io/projected/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kube-api-access-zmh96\") pod \"auth-enabled-test-kserve-75b6dc484f-q8968\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.379051 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.379018    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:46.397844 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.397815    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:46.681254 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.681229    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"]
Sep 16 13:06:46.687579 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:06:46.687545    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadaa74df_2cc6_47d8_b5a8_ec835619c8e0.slice/crio-7f1f0d6277d0e92d5bc9933f53011cc7000122b1693b1bf58027894345c37d0e WatchSource:0}: Error finding container 7f1f0d6277d0e92d5bc9933f53011cc7000122b1693b1bf58027894345c37d0e: Status 404 returned error can't find the container with id 7f1f0d6277d0e92d5bc9933f53011cc7000122b1693b1bf58027894345c37d0e
Sep 16 13:06:46.787573 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.787532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" event={"ID":"adaa74df-2cc6-47d8-b5a8-ec835619c8e0","Type":"ContainerStarted","Data":"a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b"}
Sep 16 13:06:46.787731 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:46.787588    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" event={"ID":"adaa74df-2cc6-47d8-b5a8-ec835619c8e0","Type":"ContainerStarted","Data":"7f1f0d6277d0e92d5bc9933f53011cc7000122b1693b1bf58027894345c37d0e"}
Sep 16 13:06:49.804831 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:49.804795    2575 generic.go:356] "Generic (PLEG): container finished" podID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerID="a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b" exitCode=0
Sep 16 13:06:49.805190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:49.804868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" event={"ID":"adaa74df-2cc6-47d8-b5a8-ec835619c8e0","Type":"ContainerDied","Data":"a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b"}
Sep 16 13:06:50.812113 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:50.812080    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" event={"ID":"adaa74df-2cc6-47d8-b5a8-ec835619c8e0","Type":"ContainerStarted","Data":"db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab"}
Sep 16 13:06:50.839804 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:50.839754    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podStartSLOduration=2.527915346 podStartE2EDuration="4.839739027s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.311823681s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:06:50.837448327 +0000 UTC m=+1132.018590989" watchObservedRunningTime="2026-09-16 13:06:50.839739027 +0000 UTC m=+1132.020881684"
Sep 16 13:06:51.398345 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:51.398319    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:53.114574 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:53.114524    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:53.214708 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:53.214669    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:06:53.452479 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:53.452402    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:06:53.460263 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:53.460233    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:06:56.398962 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:56.398925    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:06:56.400540 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:56.400510    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:06:57.068484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:06:57.068449    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:07:04.171880 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:04.171841    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:07:04.177885 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:04.177856    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"]
Sep 16 13:07:04.189539 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:04.189204    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 13:07:04.270450 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:04.270398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:07:06.398872 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:06.398829    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:07:08.274427 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:08.274387    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:07:16.398897 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:16.398853    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:07:18.274784 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:18.274740    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:07:19.203981 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.203942    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" containerName="main" containerID="cri-o://039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea" gracePeriod=45
Sep 16 13:07:19.212238 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.212205    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" containerID="cri-o://20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc" gracePeriod=45
Sep 16 13:07:19.453526 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.453473    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 16 13:07:19.961034 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.961003    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:07:19.988682 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.988632    2575 generic.go:356] "Generic (PLEG): container finished" podID="09f692da-4756-4e99-8fb4-33dd892fc035" containerID="20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc" exitCode=0
Sep 16 13:07:19.988849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.988706    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09f692da-4756-4e99-8fb4-33dd892fc035","Type":"ContainerDied","Data":"20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc"}
Sep 16 13:07:19.988849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.988757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09f692da-4756-4e99-8fb4-33dd892fc035","Type":"ContainerDied","Data":"974546ca5100d7d0b01f538772d485a13639171f608464bb64a3795e0c415f78"}
Sep 16 13:07:19.988849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.988789    2575 scope.go:117] "RemoveContainer" containerID="20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc"
Sep 16 13:07:19.989034 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.988923    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 13:07:19.999837 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:19.999815    2575 scope.go:117] "RemoveContainer" containerID="b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f"
Sep 16 13:07:20.009840 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.009817    2575 scope.go:117] "RemoveContainer" containerID="20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc"
Sep 16 13:07:20.010114 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:07:20.010095    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc\": container with ID starting with 20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc not found: ID does not exist" containerID="20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc"
Sep 16 13:07:20.010202 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.010125    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc"} err="failed to get container status \"20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc\": rpc error: code = NotFound desc = could not find container \"20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc\": container with ID starting with 20256ce07924b1bb7c00e034f86be36d7cc6b0f464a1e824d2009522d240d6fc not found: ID does not exist"
Sep 16 13:07:20.010202 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.010149    2575 scope.go:117] "RemoveContainer" containerID="b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f"
Sep 16 13:07:20.010472 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:07:20.010445    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f\": container with ID starting with b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f not found: ID does not exist" containerID="b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f"
Sep 16 13:07:20.010530 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.010469    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f"} err="failed to get container status \"b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f\": rpc error: code = NotFound desc = could not find container \"b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f\": container with ID starting with b445c198d0bdcf6047a6eb1c3f15ca6685689b1d25bf489ba73e4fd616575c1f not found: ID does not exist"
Sep 16 13:07:20.104536 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.104511    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f692da-4756-4e99-8fb4-33dd892fc035-tls-certs\") pod \"09f692da-4756-4e99-8fb4-33dd892fc035\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") "
Sep 16 13:07:20.104713 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.104562    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-kserve-provision-location\") pod \"09f692da-4756-4e99-8fb4-33dd892fc035\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") "
Sep 16 13:07:20.104713 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.104591    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-tmp-dir\") pod \"09f692da-4756-4e99-8fb4-33dd892fc035\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") "
Sep 16 13:07:20.104713 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.104620    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-home\") pod \"09f692da-4756-4e99-8fb4-33dd892fc035\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") "
Sep 16 13:07:20.104713 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.104680    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mngp\" (UniqueName: \"kubernetes.io/projected/09f692da-4756-4e99-8fb4-33dd892fc035-kube-api-access-4mngp\") pod \"09f692da-4756-4e99-8fb4-33dd892fc035\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") "
Sep 16 13:07:20.104953 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.104725    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-dshm\") pod \"09f692da-4756-4e99-8fb4-33dd892fc035\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") "
Sep 16 13:07:20.104953 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.104769    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f692da-4756-4e99-8fb4-33dd892fc035-cabundle-cert\") pod \"09f692da-4756-4e99-8fb4-33dd892fc035\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") "
Sep 16 13:07:20.105149 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.105129    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-model-cache\") pod \"09f692da-4756-4e99-8fb4-33dd892fc035\" (UID: \"09f692da-4756-4e99-8fb4-33dd892fc035\") "
Sep 16 13:07:20.105258 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.105228    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-home" (OuterVolumeSpecName: "home") pod "09f692da-4756-4e99-8fb4-33dd892fc035" (UID: "09f692da-4756-4e99-8fb4-33dd892fc035"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:20.105874 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.105660    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f692da-4756-4e99-8fb4-33dd892fc035-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09f692da-4756-4e99-8fb4-33dd892fc035" (UID: "09f692da-4756-4e99-8fb4-33dd892fc035"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:07:20.106054 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.106020    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-model-cache" (OuterVolumeSpecName: "model-cache") pod "09f692da-4756-4e99-8fb4-33dd892fc035" (UID: "09f692da-4756-4e99-8fb4-33dd892fc035"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:20.106320 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.106302    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:20.106415 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.106325    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09f692da-4756-4e99-8fb4-33dd892fc035-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:20.106415 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.106339    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:20.107348 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.107315    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-dshm" (OuterVolumeSpecName: "dshm") pod "09f692da-4756-4e99-8fb4-33dd892fc035" (UID: "09f692da-4756-4e99-8fb4-33dd892fc035"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:20.107516 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.107492    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f692da-4756-4e99-8fb4-33dd892fc035-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09f692da-4756-4e99-8fb4-33dd892fc035" (UID: "09f692da-4756-4e99-8fb4-33dd892fc035"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:07:20.107684 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.107635    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f692da-4756-4e99-8fb4-33dd892fc035-kube-api-access-4mngp" (OuterVolumeSpecName: "kube-api-access-4mngp") pod "09f692da-4756-4e99-8fb4-33dd892fc035" (UID: "09f692da-4756-4e99-8fb4-33dd892fc035"). InnerVolumeSpecName "kube-api-access-4mngp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:07:20.118002 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.117981    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09f692da-4756-4e99-8fb4-33dd892fc035" (UID: "09f692da-4756-4e99-8fb4-33dd892fc035"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:20.136273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.136242    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09f692da-4756-4e99-8fb4-33dd892fc035" (UID: "09f692da-4756-4e99-8fb4-33dd892fc035"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:07:20.207589 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.207562    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09f692da-4756-4e99-8fb4-33dd892fc035-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:20.207589 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.207591    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:20.207765 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.207602    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:20.207765 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.207612    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mngp\" (UniqueName: \"kubernetes.io/projected/09f692da-4756-4e99-8fb4-33dd892fc035-kube-api-access-4mngp\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:20.207765 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.207620    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09f692da-4756-4e99-8fb4-33dd892fc035-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:20.315205 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.315176    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 13:07:20.318038 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:20.318015    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 13:07:21.330595 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:21.330556    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" path="/var/lib/kubelet/pods/09f692da-4756-4e99-8fb4-33dd892fc035/volumes"
Sep 16 13:07:26.398182 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:26.398141    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:07:28.274505 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:28.274464    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:07:28.274984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:28.274592    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:07:36.398503 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:36.398455    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:07:38.274948 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:38.274906    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:07:46.399098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:46.399060    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:07:48.274939 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:48.274901    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:07:49.453434 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:49.453413    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:07:49.548486 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:49.548410    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99q89\" (UniqueName: \"kubernetes.io/projected/6611f009-f4b0-4df7-bd52-0862a117340a-kube-api-access-99q89\") pod \"6611f009-f4b0-4df7-bd52-0862a117340a\" (UID: \"6611f009-f4b0-4df7-bd52-0862a117340a\") "
Sep 16 13:07:49.548625 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:49.548483    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6611f009-f4b0-4df7-bd52-0862a117340a-tls-certs\") pod \"6611f009-f4b0-4df7-bd52-0862a117340a\" (UID: \"6611f009-f4b0-4df7-bd52-0862a117340a\") "
Sep 16 13:07:49.550683 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:49.550629    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6611f009-f4b0-4df7-bd52-0862a117340a-kube-api-access-99q89" (OuterVolumeSpecName: "kube-api-access-99q89") pod "6611f009-f4b0-4df7-bd52-0862a117340a" (UID: "6611f009-f4b0-4df7-bd52-0862a117340a"). InnerVolumeSpecName "kube-api-access-99q89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:07:49.550785 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:49.550632    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6611f009-f4b0-4df7-bd52-0862a117340a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6611f009-f4b0-4df7-bd52-0862a117340a" (UID: "6611f009-f4b0-4df7-bd52-0862a117340a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:07:49.649765 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:49.649737    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99q89\" (UniqueName: \"kubernetes.io/projected/6611f009-f4b0-4df7-bd52-0862a117340a-kube-api-access-99q89\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:49.649765 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:49.649762    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6611f009-f4b0-4df7-bd52-0862a117340a-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:07:50.152479 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:50.152443    2575 generic.go:356] "Generic (PLEG): container finished" podID="6611f009-f4b0-4df7-bd52-0862a117340a" containerID="039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea" exitCode=0
Sep 16 13:07:50.152692 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:50.152492    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"
Sep 16 13:07:50.152692 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:50.152528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" event={"ID":"6611f009-f4b0-4df7-bd52-0862a117340a","Type":"ContainerDied","Data":"039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea"}
Sep 16 13:07:50.152692 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:50.152591    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d" event={"ID":"6611f009-f4b0-4df7-bd52-0862a117340a","Type":"ContainerDied","Data":"2e07898618d4af2caddefbf3a2bb9d6e2b1dcd1f3aaafa131d4c8af7b49fe5fa"}
Sep 16 13:07:50.152692 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:50.152621    2575 scope.go:117] "RemoveContainer" containerID="039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea"
Sep 16 13:07:50.162187 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:50.162172    2575 scope.go:117] "RemoveContainer" containerID="039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea"
Sep 16 13:07:50.162450 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:07:50.162434    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea\": container with ID starting with 039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea not found: ID does not exist" containerID="039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea"
Sep 16 13:07:50.162512 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:50.162460    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea"} err="failed to get container status \"039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea\": rpc error: code = NotFound desc = could not find container \"039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea\": container with ID starting with 039d38e23458c8e3d6fb1610cf9b5b37b2a72b650996d845d7ef6cd42391b0ea not found: ID does not exist"
Sep 16 13:07:50.174258 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:50.174235    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"]
Sep 16 13:07:50.177226 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:50.177204    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schen2f2d"]
Sep 16 13:07:51.329412 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:51.329382    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" path="/var/lib/kubelet/pods/6611f009-f4b0-4df7-bd52-0862a117340a/volumes"
Sep 16 13:07:54.111707 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:54.111672    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:07:54.207706 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:54.207670    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:07:56.398823 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:07:56.398785    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:08:00.527016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:00.526961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:06.240819 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:06.240790    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:06.341812 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:06.341780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:06.398469 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:06.398407    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:08:07.313596 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.313559    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:07.474259 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.474225    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"]
Sep 16 13:08:07.474632 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.474617    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" containerName="main"
Sep 16 13:08:07.474733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.474634    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" containerName="main"
Sep 16 13:08:07.474733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.474671    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="storage-initializer"
Sep 16 13:08:07.474733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.474679    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="storage-initializer"
Sep 16 13:08:07.474733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.474695    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main"
Sep 16 13:08:07.474733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.474701    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main"
Sep 16 13:08:07.474890 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.474761    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6611f009-f4b0-4df7-bd52-0862a117340a" containerName="main"
Sep 16 13:08:07.474890 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.474770    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="09f692da-4756-4e99-8fb4-33dd892fc035" containerName="main"
Sep 16 13:08:07.479074 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.479056    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:07.483998 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.483975    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-pkg56\""
Sep 16 13:08:07.483998 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.483988    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-fz5lj\""
Sep 16 13:08:07.484178 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.484082    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 13:08:07.484565 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.484280    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 13:08:07.484565 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.484293    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 13:08:07.486352 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.486325    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"]
Sep 16 13:08:07.498229 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.498198    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:07.590848 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.590759    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtbm\" (UniqueName: \"kubernetes.io/projected/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-kube-api-access-nrtbm\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj\" (UID: \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:07.590848 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.590817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj\" (UID: \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:07.692250 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.692206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj\" (UID: \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:07.692444 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.692299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtbm\" (UniqueName: \"kubernetes.io/projected/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-kube-api-access-nrtbm\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj\" (UID: \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:07.694898 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.694876    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj\" (UID: \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:07.700275 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.700256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtbm\" (UniqueName: \"kubernetes.io/projected/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-kube-api-access-nrtbm\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj\" (UID: \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:07.792818 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:07.792783    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:08.067584 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:08.067563    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"]
Sep 16 13:08:08.069243 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:08:08.069220    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2a5c3f_2a20_48fa_b136_fd15807f22e3.slice/crio-4a222feffd1d62ff0cf6642a8635e7af098cba827eabf7e47375ee88bb236586 WatchSource:0}: Error finding container 4a222feffd1d62ff0cf6642a8635e7af098cba827eabf7e47375ee88bb236586: Status 404 returned error can't find the container with id 4a222feffd1d62ff0cf6642a8635e7af098cba827eabf7e47375ee88bb236586
Sep 16 13:08:08.262332 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:08.262298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" event={"ID":"4c2a5c3f-2a20-48fa-b136-fd15807f22e3","Type":"ContainerStarted","Data":"59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7"}
Sep 16 13:08:08.262494 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:08.262355    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" event={"ID":"4c2a5c3f-2a20-48fa-b136-fd15807f22e3","Type":"ContainerStarted","Data":"4a222feffd1d62ff0cf6642a8635e7af098cba827eabf7e47375ee88bb236586"}
Sep 16 13:08:08.262494 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:08.262391    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:08.293695 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:08.293615    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" podStartSLOduration=1.293594689 podStartE2EDuration="1.293594689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:08:08.289855167 +0000 UTC m=+1209.470997817" watchObservedRunningTime="2026-09-16 13:08:08.293594689 +0000 UTC m=+1209.474737344"
Sep 16 13:08:15.147078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:15.147044    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:16.398950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:16.398906    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:08:26.399170 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:26.399109    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:08:36.398851 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:36.398809    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 13:08:39.270876 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:39.270840    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:08:40.552693 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:40.550609    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"]
Sep 16 13:08:40.562260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:40.562228    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:40.619683 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:40.619637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:46.408230 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:46.408203    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:08:46.415866 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:46.415845    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:08:49.270616 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:49.270576    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" podUID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:08:51.411283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:51.411185    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:51.419507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:51.419483    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"]
Sep 16 13:08:51.510129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:51.510095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:08:55.582906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:55.582870    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" podUID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" containerName="main" containerID="cri-o://59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7" gracePeriod=45
Sep 16 13:08:59.271180 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:08:59.271138    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" podUID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:09:03.892605 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:03.892566    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:06.443229 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:06.443160    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main" containerID="cri-o://db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab" gracePeriod=45
Sep 16 13:09:07.110094 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.110071    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:09:07.171229 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171204    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tmp-dir\") pod \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") "
Sep 16 13:09:07.171365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171240    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-dshm\") pod \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") "
Sep 16 13:09:07.171365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171262    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-model-cache\") pod \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") "
Sep 16 13:09:07.171365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171300    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmh96\" (UniqueName: \"kubernetes.io/projected/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kube-api-access-zmh96\") pod \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") "
Sep 16 13:09:07.171365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171324    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-cabundle-cert\") pod \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") "
Sep 16 13:09:07.171365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171346    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kserve-provision-location\") pod \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") "
Sep 16 13:09:07.171672 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171382    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tls-certs\") pod \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") "
Sep 16 13:09:07.171672 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171400    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-home\") pod \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\" (UID: \"adaa74df-2cc6-47d8-b5a8-ec835619c8e0\") "
Sep 16 13:09:07.171808 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171782    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "adaa74df-2cc6-47d8-b5a8-ec835619c8e0" (UID: "adaa74df-2cc6-47d8-b5a8-ec835619c8e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:07.171992 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.171969    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "adaa74df-2cc6-47d8-b5a8-ec835619c8e0" (UID: "adaa74df-2cc6-47d8-b5a8-ec835619c8e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:07.172252 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.172231    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-home" (OuterVolumeSpecName: "home") pod "adaa74df-2cc6-47d8-b5a8-ec835619c8e0" (UID: "adaa74df-2cc6-47d8-b5a8-ec835619c8e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:07.173983 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.173955    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-dshm" (OuterVolumeSpecName: "dshm") pod "adaa74df-2cc6-47d8-b5a8-ec835619c8e0" (UID: "adaa74df-2cc6-47d8-b5a8-ec835619c8e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:07.174094 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.174037    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adaa74df-2cc6-47d8-b5a8-ec835619c8e0" (UID: "adaa74df-2cc6-47d8-b5a8-ec835619c8e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:07.174094 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.174051    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kube-api-access-zmh96" (OuterVolumeSpecName: "kube-api-access-zmh96") pod "adaa74df-2cc6-47d8-b5a8-ec835619c8e0" (UID: "adaa74df-2cc6-47d8-b5a8-ec835619c8e0"). InnerVolumeSpecName "kube-api-access-zmh96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:07.189913 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.189887    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "adaa74df-2cc6-47d8-b5a8-ec835619c8e0" (UID: "adaa74df-2cc6-47d8-b5a8-ec835619c8e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:07.198919 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.198898    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "adaa74df-2cc6-47d8-b5a8-ec835619c8e0" (UID: "adaa74df-2cc6-47d8-b5a8-ec835619c8e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:07.272508 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.272481    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:07.272508 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.272502    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:07.272668 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.272531    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:07.272668 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.272540    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:07.272668 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.272547    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:07.272668 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.272556    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmh96\" (UniqueName: \"kubernetes.io/projected/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kube-api-access-zmh96\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:07.272668 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.272565    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:07.272668 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.272573    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adaa74df-2cc6-47d8-b5a8-ec835619c8e0-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:07.589718 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.589623    2575 generic.go:356] "Generic (PLEG): container finished" podID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerID="db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab" exitCode=0
Sep 16 13:09:07.589718 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.589688    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"
Sep 16 13:09:07.590187 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.589690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" event={"ID":"adaa74df-2cc6-47d8-b5a8-ec835619c8e0","Type":"ContainerDied","Data":"db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab"}
Sep 16 13:09:07.590187 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.589808    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968" event={"ID":"adaa74df-2cc6-47d8-b5a8-ec835619c8e0","Type":"ContainerDied","Data":"7f1f0d6277d0e92d5bc9933f53011cc7000122b1693b1bf58027894345c37d0e"}
Sep 16 13:09:07.590187 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.589829    2575 scope.go:117] "RemoveContainer" containerID="db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab"
Sep 16 13:09:07.598716 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.598698    2575 scope.go:117] "RemoveContainer" containerID="a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b"
Sep 16 13:09:07.606803 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.606783    2575 scope.go:117] "RemoveContainer" containerID="db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab"
Sep 16 13:09:07.607121 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:09:07.607083    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab\": container with ID starting with db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab not found: ID does not exist" containerID="db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab"
Sep 16 13:09:07.607213 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.607122    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab"} err="failed to get container status \"db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab\": rpc error: code = NotFound desc = could not find container \"db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab\": container with ID starting with db8b2329ef405208ee09707d2f53a23c8cbe33c8fadb7ddb84d5e2b000077dab not found: ID does not exist"
Sep 16 13:09:07.607213 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.607145    2575 scope.go:117] "RemoveContainer" containerID="a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b"
Sep 16 13:09:07.607461 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:09:07.607441    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b\": container with ID starting with a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b not found: ID does not exist" containerID="a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b"
Sep 16 13:09:07.607513 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.607467    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b"} err="failed to get container status \"a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b\": rpc error: code = NotFound desc = could not find container \"a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b\": container with ID starting with a16b2f6a5558e412ea7c9f48c1aff2daacee4297b0b898ba939fe64b8479309b not found: ID does not exist"
Sep 16 13:09:07.610077 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.610055    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"]
Sep 16 13:09:07.615296 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:07.615275    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-75b6dc484f-q8968"]
Sep 16 13:09:09.271063 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:09.271030    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" podUID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:09:09.271487 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:09.271144    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:09:09.328908 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:09.328876    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" path="/var/lib/kubelet/pods/adaa74df-2cc6-47d8-b5a8-ec835619c8e0/volumes"
Sep 16 13:09:19.270915 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:19.270880    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" podUID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:09:25.835485 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:25.835461    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:09:25.905422 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:25.905397    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-tls-certs\") pod \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\" (UID: \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\") "
Sep 16 13:09:25.905566 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:25.905439    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrtbm\" (UniqueName: \"kubernetes.io/projected/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-kube-api-access-nrtbm\") pod \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\" (UID: \"4c2a5c3f-2a20-48fa-b136-fd15807f22e3\") "
Sep 16 13:09:25.907703 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:25.907671    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c2a5c3f-2a20-48fa-b136-fd15807f22e3" (UID: "4c2a5c3f-2a20-48fa-b136-fd15807f22e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:25.907808 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:25.907748    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-kube-api-access-nrtbm" (OuterVolumeSpecName: "kube-api-access-nrtbm") pod "4c2a5c3f-2a20-48fa-b136-fd15807f22e3" (UID: "4c2a5c3f-2a20-48fa-b136-fd15807f22e3"). InnerVolumeSpecName "kube-api-access-nrtbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:26.006941 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.006904    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:26.006941 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.006934    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtbm\" (UniqueName: \"kubernetes.io/projected/4c2a5c3f-2a20-48fa-b136-fd15807f22e3-kube-api-access-nrtbm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:09:26.688911 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.688877    2575 generic.go:356] "Generic (PLEG): container finished" podID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" containerID="59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7" exitCode=0
Sep 16 13:09:26.689082 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.688918    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"
Sep 16 13:09:26.689082 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.688941    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" event={"ID":"4c2a5c3f-2a20-48fa-b136-fd15807f22e3","Type":"ContainerDied","Data":"59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7"}
Sep 16 13:09:26.689082 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.688982    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj" event={"ID":"4c2a5c3f-2a20-48fa-b136-fd15807f22e3","Type":"ContainerDied","Data":"4a222feffd1d62ff0cf6642a8635e7af098cba827eabf7e47375ee88bb236586"}
Sep 16 13:09:26.689082 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.689003    2575 scope.go:117] "RemoveContainer" containerID="59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7"
Sep 16 13:09:26.698735 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.698509    2575 scope.go:117] "RemoveContainer" containerID="59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7"
Sep 16 13:09:26.698849 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:09:26.698829    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7\": container with ID starting with 59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7 not found: ID does not exist" containerID="59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7"
Sep 16 13:09:26.698905 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.698860    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7"} err="failed to get container status \"59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7\": rpc error: code = NotFound desc = could not find container \"59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7\": container with ID starting with 59ce4021323506870f5c7f452cb4884f0a50be3746e536cd8815b354a04470d7 not found: ID does not exist"
Sep 16 13:09:26.710617 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.710592    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"]
Sep 16 13:09:26.713732 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:26.713709    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbjthjj"]
Sep 16 13:09:27.328373 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:27.328334    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" path="/var/lib/kubelet/pods/4c2a5c3f-2a20-48fa-b136-fd15807f22e3/volumes"
Sep 16 13:09:30.649143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:30.649097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:30.746844 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:30.746800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:36.665081 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.665043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:36.771640 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.771611    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"]
Sep 16 13:09:36.772170 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.772154    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="storage-initializer"
Sep 16 13:09:36.772217 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.772176    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="storage-initializer"
Sep 16 13:09:36.772217 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.772188    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main"
Sep 16 13:09:36.772217 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.772197    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main"
Sep 16 13:09:36.772315 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.772216    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" containerName="main"
Sep 16 13:09:36.772315 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.772244    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" containerName="main"
Sep 16 13:09:36.772380 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.772321    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c2a5c3f-2a20-48fa-b136-fd15807f22e3" containerName="main"
Sep 16 13:09:36.772380 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.772336    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="adaa74df-2cc6-47d8-b5a8-ec835619c8e0" containerName="main"
Sep 16 13:09:36.776034 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.776012    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.778688 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.778664    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 13:09:36.778814 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.778734    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 13:09:36.779997 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.779970    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-hngvz\""
Sep 16 13:09:36.780098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.779998    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 13:09:36.787202 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.787174    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"]
Sep 16 13:09:36.803873 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.803849    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:36.887180 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.887156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.887290 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.887189    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.887290 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.887211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.887290 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.887255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1123e2e6-aa96-4650-a1ec-1df360775b6d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.887290 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.887281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6gg\" (UniqueName: \"kubernetes.io/projected/1123e2e6-aa96-4650-a1ec-1df360775b6d-kube-api-access-wr6gg\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.887443 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.887345    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.988321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.988246    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.988321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.988277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.988321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.988301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1123e2e6-aa96-4650-a1ec-1df360775b6d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.988321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.988322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6gg\" (UniqueName: \"kubernetes.io/projected/1123e2e6-aa96-4650-a1ec-1df360775b6d-kube-api-access-wr6gg\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.988638 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.988361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.988638 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.988399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.988832 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.988793    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.988832 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.988821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.989083 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.989063    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.990954 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.990932    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.991128 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.991111    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1123e2e6-aa96-4650-a1ec-1df360775b6d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:36.994175 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.994153    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:36.996753 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:36.996732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6gg\" (UniqueName: \"kubernetes.io/projected/1123e2e6-aa96-4650-a1ec-1df360775b6d-kube-api-access-wr6gg\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:37.087773 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:37.087748    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:37.361097 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:37.361073    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"]
Sep 16 13:09:37.362896 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:09:37.362874    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1123e2e6_aa96_4650_a1ec_1df360775b6d.slice/crio-be660357965d0ff74f83a988cc31fd91a97e948bafdcba74dff050873b7579c6 WatchSource:0}: Error finding container be660357965d0ff74f83a988cc31fd91a97e948bafdcba74dff050873b7579c6: Status 404 returned error can't find the container with id be660357965d0ff74f83a988cc31fd91a97e948bafdcba74dff050873b7579c6
Sep 16 13:09:37.751235 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:37.751150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2" event={"ID":"1123e2e6-aa96-4650-a1ec-1df360775b6d","Type":"ContainerStarted","Data":"0d609838b7438d759efdf604b26b12b51133e52f8a744e8709e9fd5702789385"}
Sep 16 13:09:37.751235 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:37.751209    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2" event={"ID":"1123e2e6-aa96-4650-a1ec-1df360775b6d","Type":"ContainerStarted","Data":"be660357965d0ff74f83a988cc31fd91a97e948bafdcba74dff050873b7579c6"}
Sep 16 13:09:37.770076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:37.770020    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2" podStartSLOduration=1.770000996 podStartE2EDuration="1.770000996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:37.768816748 +0000 UTC m=+1298.949959402" watchObservedRunningTime="2026-09-16 13:09:37.770000996 +0000 UTC m=+1298.951143649"
Sep 16 13:09:38.088916 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:38.088842    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:41.386113 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:41.386075    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:42.483248 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:42.483221    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:42.839499 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:42.839423    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:42.941851 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:42.941819    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:44.020742 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:44.020702    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:47.088801 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:47.088770    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:47.101746 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:47.101725    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:47.629793 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:47.629759    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:47.813098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:47.813067    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:09:51.727628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.727593    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:51.897099 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.897070    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"]
Sep 16 13:09:51.900627 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.900607    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:09:51.903142 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.903116    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-nnm2w\""
Sep 16 13:09:51.904524 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.904504    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 13:09:51.904630 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.904573    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 13:09:51.904630 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.904573    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-8cf6p\""
Sep 16 13:09:51.904819 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.904801    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 13:09:51.909943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.909922    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"]
Sep 16 13:09:51.934843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:51.934819    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:52.014412 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.014345    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx5g4\" (UniqueName: \"kubernetes.io/projected/218248e9-8704-47a4-9cb3-ab87d97b936b-kube-api-access-xx5g4\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz\" (UID: \"218248e9-8704-47a4-9cb3-ab87d97b936b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:09:52.014412 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.014376    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218248e9-8704-47a4-9cb3-ab87d97b936b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz\" (UID: \"218248e9-8704-47a4-9cb3-ab87d97b936b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:09:52.115540 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.115514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx5g4\" (UniqueName: \"kubernetes.io/projected/218248e9-8704-47a4-9cb3-ab87d97b936b-kube-api-access-xx5g4\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz\" (UID: \"218248e9-8704-47a4-9cb3-ab87d97b936b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:09:52.115671 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.115547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218248e9-8704-47a4-9cb3-ab87d97b936b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz\" (UID: \"218248e9-8704-47a4-9cb3-ab87d97b936b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:09:52.118311 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.118290    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218248e9-8704-47a4-9cb3-ab87d97b936b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz\" (UID: \"218248e9-8704-47a4-9cb3-ab87d97b936b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:09:52.123364 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.123343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx5g4\" (UniqueName: \"kubernetes.io/projected/218248e9-8704-47a4-9cb3-ab87d97b936b-kube-api-access-xx5g4\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz\" (UID: \"218248e9-8704-47a4-9cb3-ab87d97b936b\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:09:52.212619 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.212592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:09:52.485064 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.485040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"]
Sep 16 13:09:52.486599 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:09:52.486565    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218248e9_8704_47a4_9cb3_ab87d97b936b.slice/crio-e74842d59f8bb4eaa058cf14c51b780c9a20b98628b97d2984a57830a27fd285 WatchSource:0}: Error finding container e74842d59f8bb4eaa058cf14c51b780c9a20b98628b97d2984a57830a27fd285: Status 404 returned error can't find the container with id e74842d59f8bb4eaa058cf14c51b780c9a20b98628b97d2984a57830a27fd285
Sep 16 13:09:52.832106 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.831977    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" event={"ID":"218248e9-8704-47a4-9cb3-ab87d97b936b","Type":"ContainerStarted","Data":"e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed"}
Sep 16 13:09:52.832515 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.832123    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:09:52.832515 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.832150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" event={"ID":"218248e9-8704-47a4-9cb3-ab87d97b936b","Type":"ContainerStarted","Data":"e74842d59f8bb4eaa058cf14c51b780c9a20b98628b97d2984a57830a27fd285"}
Sep 16 13:09:52.851473 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:52.851420    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" podStartSLOduration=1.851394565 podStartE2EDuration="1.851394565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:52.849714526 +0000 UTC m=+1314.030857180" watchObservedRunningTime="2026-09-16 13:09:52.851394565 +0000 UTC m=+1314.032537219"
Sep 16 13:09:53.960909 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:53.960877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:54.018435 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:54.018402    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:09:59.894147 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:09:59.894106    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:10:11.227877 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:11.227837    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:10:11.234752 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:11.234727    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"]
Sep 16 13:10:11.245456 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:10:11.245420    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:10:11.245456 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:10:11.245456 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:10:11.245456 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="0d609838b7438d759efdf604b26b12b51133e52f8a744e8709e9fd5702789385" cmd=["/bin/sleep","15"]
Sep 16 13:10:11.245845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:11.245472    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2" message="" err=<
Sep 16 13:10:11.245845 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:10:11.245845 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:10:11.245845 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:10:11.245845 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:10:11.245484    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:10:11.245845 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:10:11.245845 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:10:11.245845 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2" podUID="1123e2e6-aa96-4650-a1ec-1df360775b6d" containerName="main" containerID="cri-o://0d609838b7438d759efdf604b26b12b51133e52f8a744e8709e9fd5702789385"
Sep 16 13:10:11.245845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:11.245521    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2" podUID="1123e2e6-aa96-4650-a1ec-1df360775b6d" containerName="main" containerID="cri-o://0d609838b7438d759efdf604b26b12b51133e52f8a744e8709e9fd5702789385" gracePeriod=60
Sep 16 13:10:11.325444 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:11.325412    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:10:11.802985 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:11.802937    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2" podUID="1123e2e6-aa96-4650-a1ec-1df360775b6d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 16 13:10:11.938327 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:11.938290    2575 generic.go:356] "Generic (PLEG): container finished" podID="1123e2e6-aa96-4650-a1ec-1df360775b6d" containerID="0d609838b7438d759efdf604b26b12b51133e52f8a744e8709e9fd5702789385" exitCode=0
Sep 16 13:10:11.938487 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:11.938412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2" event={"ID":"1123e2e6-aa96-4650-a1ec-1df360775b6d","Type":"ContainerDied","Data":"0d609838b7438d759efdf604b26b12b51133e52f8a744e8709e9fd5702789385"}
Sep 16 13:10:11.993532 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:11.993510    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:10:12.185430 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.185388    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1123e2e6-aa96-4650-a1ec-1df360775b6d-tls-certs\") pod \"1123e2e6-aa96-4650-a1ec-1df360775b6d\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") "
Sep 16 13:10:12.185746 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.185442    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-model-cache\") pod \"1123e2e6-aa96-4650-a1ec-1df360775b6d\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") "
Sep 16 13:10:12.185746 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.185475    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr6gg\" (UniqueName: \"kubernetes.io/projected/1123e2e6-aa96-4650-a1ec-1df360775b6d-kube-api-access-wr6gg\") pod \"1123e2e6-aa96-4650-a1ec-1df360775b6d\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") "
Sep 16 13:10:12.185746 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.185515    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-dshm\") pod \"1123e2e6-aa96-4650-a1ec-1df360775b6d\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") "
Sep 16 13:10:12.185746 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.185627    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-home\") pod \"1123e2e6-aa96-4650-a1ec-1df360775b6d\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") "
Sep 16 13:10:12.185746 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.185714    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-tmp-dir\") pod \"1123e2e6-aa96-4650-a1ec-1df360775b6d\" (UID: \"1123e2e6-aa96-4650-a1ec-1df360775b6d\") "
Sep 16 13:10:12.186035 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.185830    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "1123e2e6-aa96-4650-a1ec-1df360775b6d" (UID: "1123e2e6-aa96-4650-a1ec-1df360775b6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:10:12.186035 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.185902    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-home" (OuterVolumeSpecName: "home") pod "1123e2e6-aa96-4650-a1ec-1df360775b6d" (UID: "1123e2e6-aa96-4650-a1ec-1df360775b6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:10:12.186035 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.185970    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1123e2e6-aa96-4650-a1ec-1df360775b6d" (UID: "1123e2e6-aa96-4650-a1ec-1df360775b6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:10:12.186171 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.186068    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:10:12.186171 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.186086    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:10:12.186171 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.186099    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:10:12.187981 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.187952    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-dshm" (OuterVolumeSpecName: "dshm") pod "1123e2e6-aa96-4650-a1ec-1df360775b6d" (UID: "1123e2e6-aa96-4650-a1ec-1df360775b6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:10:12.188080 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.188016    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1123e2e6-aa96-4650-a1ec-1df360775b6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1123e2e6-aa96-4650-a1ec-1df360775b6d" (UID: "1123e2e6-aa96-4650-a1ec-1df360775b6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:10:12.188080 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.188025    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1123e2e6-aa96-4650-a1ec-1df360775b6d-kube-api-access-wr6gg" (OuterVolumeSpecName: "kube-api-access-wr6gg") pod "1123e2e6-aa96-4650-a1ec-1df360775b6d" (UID: "1123e2e6-aa96-4650-a1ec-1df360775b6d"). InnerVolumeSpecName "kube-api-access-wr6gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:10:12.287405 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.287376    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1123e2e6-aa96-4650-a1ec-1df360775b6d-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:10:12.287714 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.287405    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr6gg\" (UniqueName: \"kubernetes.io/projected/1123e2e6-aa96-4650-a1ec-1df360775b6d-kube-api-access-wr6gg\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:10:12.287714 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.287421    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1123e2e6-aa96-4650-a1ec-1df360775b6d-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:10:12.944324 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.944299    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"
Sep 16 13:10:12.944324 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.944304    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2" event={"ID":"1123e2e6-aa96-4650-a1ec-1df360775b6d","Type":"ContainerDied","Data":"be660357965d0ff74f83a988cc31fd91a97e948bafdcba74dff050873b7579c6"}
Sep 16 13:10:12.944538 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.944366    2575 scope.go:117] "RemoveContainer" containerID="0d609838b7438d759efdf604b26b12b51133e52f8a744e8709e9fd5702789385"
Sep 16 13:10:12.968275 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.968254    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"]
Sep 16 13:10:12.972282 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:12.972259    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fjn9k2"]
Sep 16 13:10:13.329452 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:13.329379    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1123e2e6-aa96-4650-a1ec-1df360775b6d" path="/var/lib/kubelet/pods/1123e2e6-aa96-4650-a1ec-1df360775b6d/volumes"
Sep 16 13:10:23.838424 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:10:23.838354    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:11:00.887722 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:00.887677    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:00.982376 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:00.982341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:07.210535 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:07.210497    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:13.061981 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:13.061945    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:13.125549 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:13.125514    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:20.287904 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.287866    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"]
Sep 16 13:11:20.288473 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.288235    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1123e2e6-aa96-4650-a1ec-1df360775b6d" containerName="main"
Sep 16 13:11:20.288473 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.288248    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1123e2e6-aa96-4650-a1ec-1df360775b6d" containerName="main"
Sep 16 13:11:20.288473 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.288317    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1123e2e6-aa96-4650-a1ec-1df360775b6d" containerName="main"
Sep 16 13:11:20.292510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.292492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.295088 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.295067    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 13:11:20.296566 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.296542    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-n9fkv\""
Sep 16 13:11:20.296697 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.296542    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 13:11:20.296781 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.296552    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 13:11:20.304516 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.304490    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"]
Sep 16 13:11:20.320589 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.320560    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:20.417853 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.417827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.417853 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.417855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.417984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.417879    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41139c-0629-4958-87d4-14ddfcff44ba-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.417984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.417904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.417984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.417929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tz46\" (UniqueName: \"kubernetes.io/projected/9c41139c-0629-4958-87d4-14ddfcff44ba-kube-api-access-8tz46\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.417984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.417962    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.516064 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.516029    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:20.519020 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.519000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.519114 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.519048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.519181 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.519161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.519219 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.519210    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41139c-0629-4958-87d4-14ddfcff44ba-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.519268 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.519233    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.519268 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.519259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tz46\" (UniqueName: \"kubernetes.io/projected/9c41139c-0629-4958-87d4-14ddfcff44ba-kube-api-access-8tz46\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.519413 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.519391    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.519474 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.519455    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.519561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.519542    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.521308 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.521286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.521789 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.521766    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41139c-0629-4958-87d4-14ddfcff44ba-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.527984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.527964    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tz46\" (UniqueName: \"kubernetes.io/projected/9c41139c-0629-4958-87d4-14ddfcff44ba-kube-api-access-8tz46\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.605021 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.604962    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:20.877213 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:20.877187    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"]
Sep 16 13:11:20.879244 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:11:20.879218    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c41139c_0629_4958_87d4_14ddfcff44ba.slice/crio-a034054e9d94adbaeed134b79b7ab18ddbd7c2b05eac7158b42f27f785753bbe WatchSource:0}: Error finding container a034054e9d94adbaeed134b79b7ab18ddbd7c2b05eac7158b42f27f785753bbe: Status 404 returned error can't find the container with id a034054e9d94adbaeed134b79b7ab18ddbd7c2b05eac7158b42f27f785753bbe
Sep 16 13:11:21.295088 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:21.294990    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h" event={"ID":"9c41139c-0629-4958-87d4-14ddfcff44ba","Type":"ContainerStarted","Data":"4d4b0fcf6ab42612dc3f029d8801aa86270ea297f6f9c3955be48ed6c3f03620"}
Sep 16 13:11:21.295088 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:21.295045    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h" event={"ID":"9c41139c-0629-4958-87d4-14ddfcff44ba","Type":"ContainerStarted","Data":"a034054e9d94adbaeed134b79b7ab18ddbd7c2b05eac7158b42f27f785753bbe"}
Sep 16 13:11:21.315150 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:21.315104    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h" podStartSLOduration=1.3150903600000001 podStartE2EDuration="1.31509036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:11:21.312690531 +0000 UTC m=+1402.493833185" watchObservedRunningTime="2026-09-16 13:11:21.31509036 +0000 UTC m=+1402.496233013"
Sep 16 13:11:21.605324 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:21.605253    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:30.605336 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:30.605301    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:30.617703 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:30.617676    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:31.362136 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:31.362107    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:31.677824 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:31.677739    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:54.169315 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.169236    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"]
Sep 16 13:11:54.178947 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:11:54.178913    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:11:54.178947 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:11:54.178947 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:11:54.178947 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="4d4b0fcf6ab42612dc3f029d8801aa86270ea297f6f9c3955be48ed6c3f03620" cmd=["/bin/sleep","15"]
Sep 16 13:11:54.179182 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.178971    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h" message="" err=<
Sep 16 13:11:54.179182 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:11:54.179182 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:11:54.179182 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:11:54.179182 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:11:54.178983    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:11:54.179182 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:11:54.179182 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:11:54.179182 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h" podUID="9c41139c-0629-4958-87d4-14ddfcff44ba" containerName="main" containerID="cri-o://4d4b0fcf6ab42612dc3f029d8801aa86270ea297f6f9c3955be48ed6c3f03620"
Sep 16 13:11:54.179182 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.179017    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h" podUID="9c41139c-0629-4958-87d4-14ddfcff44ba" containerName="main" containerID="cri-o://4d4b0fcf6ab42612dc3f029d8801aa86270ea297f6f9c3955be48ed6c3f03620" gracePeriod=60
Sep 16 13:11:54.196419 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.196384    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:54.274674 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.274626    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:54.351967 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.351923    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h" podUID="9c41139c-0629-4958-87d4-14ddfcff44ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 16 13:11:54.469770 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.469697    2575 generic.go:356] "Generic (PLEG): container finished" podID="9c41139c-0629-4958-87d4-14ddfcff44ba" containerID="4d4b0fcf6ab42612dc3f029d8801aa86270ea297f6f9c3955be48ed6c3f03620" exitCode=0
Sep 16 13:11:54.469902 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.469775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h" event={"ID":"9c41139c-0629-4958-87d4-14ddfcff44ba","Type":"ContainerDied","Data":"4d4b0fcf6ab42612dc3f029d8801aa86270ea297f6f9c3955be48ed6c3f03620"}
Sep 16 13:11:54.924187 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.924112    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:54.992106 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992078    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tz46\" (UniqueName: \"kubernetes.io/projected/9c41139c-0629-4958-87d4-14ddfcff44ba-kube-api-access-8tz46\") pod \"9c41139c-0629-4958-87d4-14ddfcff44ba\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") "
Sep 16 13:11:54.992275 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992132    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-home\") pod \"9c41139c-0629-4958-87d4-14ddfcff44ba\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") "
Sep 16 13:11:54.992275 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992164    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-tmp-dir\") pod \"9c41139c-0629-4958-87d4-14ddfcff44ba\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") "
Sep 16 13:11:54.992275 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992190    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-dshm\") pod \"9c41139c-0629-4958-87d4-14ddfcff44ba\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") "
Sep 16 13:11:54.992275 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992229    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-model-cache\") pod \"9c41139c-0629-4958-87d4-14ddfcff44ba\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") "
Sep 16 13:11:54.992481 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992292    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41139c-0629-4958-87d4-14ddfcff44ba-tls-certs\") pod \"9c41139c-0629-4958-87d4-14ddfcff44ba\" (UID: \"9c41139c-0629-4958-87d4-14ddfcff44ba\") "
Sep 16 13:11:54.992481 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992430    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-home" (OuterVolumeSpecName: "home") pod "9c41139c-0629-4958-87d4-14ddfcff44ba" (UID: "9c41139c-0629-4958-87d4-14ddfcff44ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:54.992481 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992445    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c41139c-0629-4958-87d4-14ddfcff44ba" (UID: "9c41139c-0629-4958-87d4-14ddfcff44ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:54.992637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992543    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c41139c-0629-4958-87d4-14ddfcff44ba" (UID: "9c41139c-0629-4958-87d4-14ddfcff44ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:54.992637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992572    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:11:54.992637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.992613    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:11:54.994457 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.994429    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-dshm" (OuterVolumeSpecName: "dshm") pod "9c41139c-0629-4958-87d4-14ddfcff44ba" (UID: "9c41139c-0629-4958-87d4-14ddfcff44ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:54.994552 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.994522    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c41139c-0629-4958-87d4-14ddfcff44ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c41139c-0629-4958-87d4-14ddfcff44ba" (UID: "9c41139c-0629-4958-87d4-14ddfcff44ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:54.994603 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:54.994587    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c41139c-0629-4958-87d4-14ddfcff44ba-kube-api-access-8tz46" (OuterVolumeSpecName: "kube-api-access-8tz46") pod "9c41139c-0629-4958-87d4-14ddfcff44ba" (UID: "9c41139c-0629-4958-87d4-14ddfcff44ba"). InnerVolumeSpecName "kube-api-access-8tz46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:11:55.093527 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.093430    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c41139c-0629-4958-87d4-14ddfcff44ba-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:11:55.093527 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.093456    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tz46\" (UniqueName: \"kubernetes.io/projected/9c41139c-0629-4958-87d4-14ddfcff44ba-kube-api-access-8tz46\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:11:55.093527 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.093471    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:11:55.093527 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.093481    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c41139c-0629-4958-87d4-14ddfcff44ba-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:11:55.119665 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.119612    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"]
Sep 16 13:11:55.124884 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.124839    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:55.185047 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.184990    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:11:55.476912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.476884    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"
Sep 16 13:11:55.477103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.476887    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h" event={"ID":"9c41139c-0629-4958-87d4-14ddfcff44ba","Type":"ContainerDied","Data":"a034054e9d94adbaeed134b79b7ab18ddbd7c2b05eac7158b42f27f785753bbe"}
Sep 16 13:11:55.477103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.477015    2575 scope.go:117] "RemoveContainer" containerID="4d4b0fcf6ab42612dc3f029d8801aa86270ea297f6f9c3955be48ed6c3f03620"
Sep 16 13:11:55.494485 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.494465    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"]
Sep 16 13:11:55.496898 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:55.496881    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4nz9h"]
Sep 16 13:11:57.329730 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:11:57.329696    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c41139c-0629-4958-87d4-14ddfcff44ba" path="/var/lib/kubelet/pods/9c41139c-0629-4958-87d4-14ddfcff44ba/volumes"
Sep 16 13:12:03.838696 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:03.838662    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" podUID="218248e9-8704-47a4-9cb3-ab87d97b936b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:12:04.165323 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:04.165296    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:10.141268 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:10.141200    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" podUID="218248e9-8704-47a4-9cb3-ab87d97b936b" containerName="main" containerID="cri-o://e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed" gracePeriod=45
Sep 16 13:12:13.838917 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:13.838884    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" podUID="218248e9-8704-47a4-9cb3-ab87d97b936b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:12:23.838765 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:23.838727    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" podUID="218248e9-8704-47a4-9cb3-ab87d97b936b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:12:23.839128 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:23.838856    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:12:33.838859 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:33.838822    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" podUID="218248e9-8704-47a4-9cb3-ab87d97b936b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:12:40.382816 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.382790    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:12:40.432156 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.432095    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx5g4\" (UniqueName: \"kubernetes.io/projected/218248e9-8704-47a4-9cb3-ab87d97b936b-kube-api-access-xx5g4\") pod \"218248e9-8704-47a4-9cb3-ab87d97b936b\" (UID: \"218248e9-8704-47a4-9cb3-ab87d97b936b\") "
Sep 16 13:12:40.432156 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.432146    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218248e9-8704-47a4-9cb3-ab87d97b936b-tls-certs\") pod \"218248e9-8704-47a4-9cb3-ab87d97b936b\" (UID: \"218248e9-8704-47a4-9cb3-ab87d97b936b\") "
Sep 16 13:12:40.434402 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.434369    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218248e9-8704-47a4-9cb3-ab87d97b936b-kube-api-access-xx5g4" (OuterVolumeSpecName: "kube-api-access-xx5g4") pod "218248e9-8704-47a4-9cb3-ab87d97b936b" (UID: "218248e9-8704-47a4-9cb3-ab87d97b936b"). InnerVolumeSpecName "kube-api-access-xx5g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:12:40.434402 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.434386    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218248e9-8704-47a4-9cb3-ab87d97b936b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "218248e9-8704-47a4-9cb3-ab87d97b936b" (UID: "218248e9-8704-47a4-9cb3-ab87d97b936b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:12:40.532678 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.532631    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx5g4\" (UniqueName: \"kubernetes.io/projected/218248e9-8704-47a4-9cb3-ab87d97b936b-kube-api-access-xx5g4\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:12:40.532678 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.532673    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/218248e9-8704-47a4-9cb3-ab87d97b936b-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:12:40.701028 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.700950    2575 generic.go:356] "Generic (PLEG): container finished" podID="218248e9-8704-47a4-9cb3-ab87d97b936b" containerID="e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed" exitCode=0
Sep 16 13:12:40.701028 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.700991    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"
Sep 16 13:12:40.701028 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.701013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" event={"ID":"218248e9-8704-47a4-9cb3-ab87d97b936b","Type":"ContainerDied","Data":"e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed"}
Sep 16 13:12:40.701253 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.701060    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz" event={"ID":"218248e9-8704-47a4-9cb3-ab87d97b936b","Type":"ContainerDied","Data":"e74842d59f8bb4eaa058cf14c51b780c9a20b98628b97d2984a57830a27fd285"}
Sep 16 13:12:40.701253 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.701086    2575 scope.go:117] "RemoveContainer" containerID="e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed"
Sep 16 13:12:40.713325 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.713305    2575 scope.go:117] "RemoveContainer" containerID="e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed"
Sep 16 13:12:40.713613 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:12:40.713582    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed\": container with ID starting with e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed not found: ID does not exist" containerID="e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed"
Sep 16 13:12:40.713740 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.713621    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed"} err="failed to get container status \"e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed\": rpc error: code = NotFound desc = could not find container \"e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed\": container with ID starting with e1fcba2082b114a793eda41f29ad6957638c6b4ff593abaf539f9a25a9c9abed not found: ID does not exist"
Sep 16 13:12:40.723949 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.723928    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"]
Sep 16 13:12:40.729392 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:40.729368    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9crf4zz"]
Sep 16 13:12:41.329338 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:41.329308    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="218248e9-8704-47a4-9cb3-ab87d97b936b" path="/var/lib/kubelet/pods/218248e9-8704-47a4-9cb3-ab87d97b936b/volumes"
Sep 16 13:12:43.802022 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:43.801973    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:43.902271 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:43.902232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:44.001273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:44.001248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:44.314688 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:44.314633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:44.411168 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:44.411135    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:48.760978 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:48.760943    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:50.310984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:50.310950    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:50.813176 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:50.813139    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:55.892893 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:55.892860    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:55.992872 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:55.992833    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:56.473427 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:56.473397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:12:56.543455 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:12:56.543407    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:06.401047 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.401017    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"]
Sep 16 13:13:06.401510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.401368    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c41139c-0629-4958-87d4-14ddfcff44ba" containerName="main"
Sep 16 13:13:06.401510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.401380    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c41139c-0629-4958-87d4-14ddfcff44ba" containerName="main"
Sep 16 13:13:06.401510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.401394    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="218248e9-8704-47a4-9cb3-ab87d97b936b" containerName="main"
Sep 16 13:13:06.401510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.401399    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="218248e9-8704-47a4-9cb3-ab87d97b936b" containerName="main"
Sep 16 13:13:06.401510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.401463    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c41139c-0629-4958-87d4-14ddfcff44ba" containerName="main"
Sep 16 13:13:06.401510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.401471    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="218248e9-8704-47a4-9cb3-ab87d97b936b" containerName="main"
Sep 16 13:13:06.405436 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.405420    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.408511 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.408487    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 13:13:06.408511 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.408501    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 13:13:06.409682 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.409662    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-28tjj\""
Sep 16 13:13:06.409805 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.409711    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 13:13:06.415702 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.415681    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"]
Sep 16 13:13:06.426443 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.426419    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:06.540569 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.540540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.540761 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.540582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.540761 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.540690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.540761 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.540714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kq7x\" (UniqueName: \"kubernetes.io/projected/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-kube-api-access-7kq7x\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.540761 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.540745    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.540944 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.540767    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.641433 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.641404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.641570 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.641437    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.641570 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.641474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.641570 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.641492    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kq7x\" (UniqueName: \"kubernetes.io/projected/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-kube-api-access-7kq7x\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.641570 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.641522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.641570 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.641546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.641883 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.641810    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.641926 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.641908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.642037 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.641969    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.643763 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.643743    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.643999 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.643983    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.653487 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.653434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kq7x\" (UniqueName: \"kubernetes.io/projected/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-kube-api-access-7kq7x\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.681774 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.681749    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"]
Sep 16 13:13:06.685640 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.685623    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:06.688601 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.688579    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-cvqls\""
Sep 16 13:13:06.697451 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.697409    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"]
Sep 16 13:13:06.698297 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.698274    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:06.715296 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.715270    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:06.843483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.843455    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c80de0-eb10-4c9f-ab0b-3616f87726de-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r\" (UID: \"27c80de0-eb10-4c9f-ab0b-3616f87726de\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:06.843636 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.843528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdxt\" (UniqueName: \"kubernetes.io/projected/27c80de0-eb10-4c9f-ab0b-3616f87726de-kube-api-access-7jdxt\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r\" (UID: \"27c80de0-eb10-4c9f-ab0b-3616f87726de\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:06.944253 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.944184    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c80de0-eb10-4c9f-ab0b-3616f87726de-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r\" (UID: \"27c80de0-eb10-4c9f-ab0b-3616f87726de\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:06.944407 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.944269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdxt\" (UniqueName: \"kubernetes.io/projected/27c80de0-eb10-4c9f-ab0b-3616f87726de-kube-api-access-7jdxt\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r\" (UID: \"27c80de0-eb10-4c9f-ab0b-3616f87726de\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:06.946887 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.946866    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c80de0-eb10-4c9f-ab0b-3616f87726de-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r\" (UID: \"27c80de0-eb10-4c9f-ab0b-3616f87726de\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:06.953162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.953139    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdxt\" (UniqueName: \"kubernetes.io/projected/27c80de0-eb10-4c9f-ab0b-3616f87726de-kube-api-access-7jdxt\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r\" (UID: \"27c80de0-eb10-4c9f-ab0b-3616f87726de\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:06.990093 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.990071    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"]
Sep 16 13:13:06.992017 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:13:06.991991    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee74ee6_9140_4c9e_a3e5_7424f4e9cb4e.slice/crio-df06eb5a914bcc23d9d232542e7b06a4db0453c331a8ea273c39153648b8a5a6 WatchSource:0}: Error finding container df06eb5a914bcc23d9d232542e7b06a4db0453c331a8ea273c39153648b8a5a6: Status 404 returned error can't find the container with id df06eb5a914bcc23d9d232542e7b06a4db0453c331a8ea273c39153648b8a5a6
Sep 16 13:13:06.995636 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:06.995620    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:07.278159 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:07.278119    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"]
Sep 16 13:13:07.280711 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:13:07.280687    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c80de0_eb10_4c9f_ab0b_3616f87726de.slice/crio-38392744031a568b2292a9f0fe1b2f492ddac93d0b7ebecadf4591ac7f33a7b4 WatchSource:0}: Error finding container 38392744031a568b2292a9f0fe1b2f492ddac93d0b7ebecadf4591ac7f33a7b4: Status 404 returned error can't find the container with id 38392744031a568b2292a9f0fe1b2f492ddac93d0b7ebecadf4591ac7f33a7b4
Sep 16 13:13:07.839766 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:07.839735    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5" event={"ID":"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e","Type":"ContainerStarted","Data":"a561b50085869cbcf28370c05fa7dc8595ec9e93b3dc4703fe40d5fc7d2ee3e8"}
Sep 16 13:13:07.840380 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:07.839792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5" event={"ID":"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e","Type":"ContainerStarted","Data":"df06eb5a914bcc23d9d232542e7b06a4db0453c331a8ea273c39153648b8a5a6"}
Sep 16 13:13:07.841772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:07.841740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r" event={"ID":"27c80de0-eb10-4c9f-ab0b-3616f87726de","Type":"ContainerStarted","Data":"089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1"}
Sep 16 13:13:07.841913 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:07.841794    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r" event={"ID":"27c80de0-eb10-4c9f-ab0b-3616f87726de","Type":"ContainerStarted","Data":"38392744031a568b2292a9f0fe1b2f492ddac93d0b7ebecadf4591ac7f33a7b4"}
Sep 16 13:13:07.841913 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:07.841896    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:07.858451 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:07.858398    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5" podStartSLOduration=1.858384516 podStartE2EDuration="1.858384516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:07.856711162 +0000 UTC m=+1509.037853816" watchObservedRunningTime="2026-09-16 13:13:07.858384516 +0000 UTC m=+1509.039527169"
Sep 16 13:13:07.872443 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:07.872400    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r" podStartSLOduration=1.8723865499999999 podStartE2EDuration="1.87238655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:13:07.872026875 +0000 UTC m=+1509.053169524" watchObservedRunningTime="2026-09-16 13:13:07.87238655 +0000 UTC m=+1509.053529203"
Sep 16 13:13:08.715658 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:08.715601    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:16.716114 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:16.716070    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:16.728618 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:16.728598    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:16.901146 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:16.901121    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:31.017598 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:31.017518    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:31.117122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:31.117095    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:34.490905 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:34.490876    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:38.849433 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:38.849399    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:13:40.977796 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:40.977766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:41.085771 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:41.085743    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:51.778492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:51.778447    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:54.909445 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:54.909408    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:54.918931 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:54.918906    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"]
Sep 16 13:13:54.925306 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:54.924983    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"]
Sep 16 13:13:54.935318 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:13:54.935284    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:13:54.935318 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:13:54.935318 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:13:54.935318 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="a561b50085869cbcf28370c05fa7dc8595ec9e93b3dc4703fe40d5fc7d2ee3e8" cmd=["/bin/sleep","15"]
Sep 16 13:13:54.935545 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:54.935338    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5" message="" err=<
Sep 16 13:13:54.935545 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:13:54.935545 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:13:54.935545 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:13:54.935545 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:13:54.935351    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:13:54.935545 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:13:54.935545 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:13:54.935545 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5" podUID="6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" containerName="main" containerID="cri-o://a561b50085869cbcf28370c05fa7dc8595ec9e93b3dc4703fe40d5fc7d2ee3e8"
Sep 16 13:13:54.935545 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:54.935396    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5" podUID="6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" containerName="main" containerID="cri-o://a561b50085869cbcf28370c05fa7dc8595ec9e93b3dc4703fe40d5fc7d2ee3e8" gracePeriod=60
Sep 16 13:13:55.001912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.001884    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:13:55.089213 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.089185    2575 generic.go:356] "Generic (PLEG): container finished" podID="6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" containerID="a561b50085869cbcf28370c05fa7dc8595ec9e93b3dc4703fe40d5fc7d2ee3e8" exitCode=0
Sep 16 13:13:55.089319 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.089265    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5" event={"ID":"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e","Type":"ContainerDied","Data":"a561b50085869cbcf28370c05fa7dc8595ec9e93b3dc4703fe40d5fc7d2ee3e8"}
Sep 16 13:13:55.683263 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.683235    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:55.700843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.700822    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-model-cache\") pod \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") "
Sep 16 13:13:55.700947 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.700855    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tmp-dir\") pod \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") "
Sep 16 13:13:55.700947 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.700884    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-home\") pod \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") "
Sep 16 13:13:55.700947 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.700928    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-dshm\") pod \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") "
Sep 16 13:13:55.701117 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.700965    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kq7x\" (UniqueName: \"kubernetes.io/projected/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-kube-api-access-7kq7x\") pod \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") "
Sep 16 13:13:55.701117 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.700999    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tls-certs\") pod \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\" (UID: \"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e\") "
Sep 16 13:13:55.701223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.701099    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" (UID: "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:13:55.701223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.701178    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-home" (OuterVolumeSpecName: "home") pod "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" (UID: "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:13:55.701223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.701109    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" (UID: "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:13:55.701372 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.701349    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:13:55.701372 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.701367    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:13:55.701449 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.701382    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:13:55.703352 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.703328    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-dshm" (OuterVolumeSpecName: "dshm") pod "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" (UID: "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:13:55.703865 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.703821    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-kube-api-access-7kq7x" (OuterVolumeSpecName: "kube-api-access-7kq7x") pod "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" (UID: "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e"). InnerVolumeSpecName "kube-api-access-7kq7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:13:55.704066 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.704044    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" (UID: "6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:13:55.802544 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.802470    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:13:55.802544 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.802495    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kq7x\" (UniqueName: \"kubernetes.io/projected/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-kube-api-access-7kq7x\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:13:55.802544 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:55.802505    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:13:56.096231 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:56.096146    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"
Sep 16 13:13:56.096231 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:56.096200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5" event={"ID":"6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e","Type":"ContainerDied","Data":"df06eb5a914bcc23d9d232542e7b06a4db0453c331a8ea273c39153648b8a5a6"}
Sep 16 13:13:56.096747 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:56.096257    2575 scope.go:117] "RemoveContainer" containerID="a561b50085869cbcf28370c05fa7dc8595ec9e93b3dc4703fe40d5fc7d2ee3e8"
Sep 16 13:13:56.121360 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:56.121338    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"]
Sep 16 13:13:56.127063 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:56.127038    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-tqzd5"]
Sep 16 13:13:57.330415 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:57.330380    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" path="/var/lib/kubelet/pods/6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e/volumes"
Sep 16 13:13:58.848839 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:13:58.848800    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r" podUID="27c80de0-eb10-4c9f-ab0b-3616f87726de" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:14:05.612410 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.611978    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 13:14:05.612410 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:14:05.612026    2575 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r" podUID="27c80de0-eb10-4c9f-ab0b-3616f87726de" containerName="main" containerID="cri-o://089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1"
Sep 16 13:14:05.612410 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.612093    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r" podUID="27c80de0-eb10-4c9f-ab0b-3616f87726de" containerName="main" containerID="cri-o://089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1" gracePeriod=50
Sep 16 13:14:05.752493 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.752469    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r_27c80de0-eb10-4c9f-ab0b-3616f87726de/main/0.log"
Sep 16 13:14:05.754720 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.754705    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:14:05.783281 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.783259    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jdxt\" (UniqueName: \"kubernetes.io/projected/27c80de0-eb10-4c9f-ab0b-3616f87726de-kube-api-access-7jdxt\") pod \"27c80de0-eb10-4c9f-ab0b-3616f87726de\" (UID: \"27c80de0-eb10-4c9f-ab0b-3616f87726de\") "
Sep 16 13:14:05.783397 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.783345    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c80de0-eb10-4c9f-ab0b-3616f87726de-tls-certs\") pod \"27c80de0-eb10-4c9f-ab0b-3616f87726de\" (UID: \"27c80de0-eb10-4c9f-ab0b-3616f87726de\") "
Sep 16 13:14:05.785464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.785436    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c80de0-eb10-4c9f-ab0b-3616f87726de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27c80de0-eb10-4c9f-ab0b-3616f87726de" (UID: "27c80de0-eb10-4c9f-ab0b-3616f87726de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:14:05.785581 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.785469    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c80de0-eb10-4c9f-ab0b-3616f87726de-kube-api-access-7jdxt" (OuterVolumeSpecName: "kube-api-access-7jdxt") pod "27c80de0-eb10-4c9f-ab0b-3616f87726de" (UID: "27c80de0-eb10-4c9f-ab0b-3616f87726de"). InnerVolumeSpecName "kube-api-access-7jdxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:14:05.885018 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.884954    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jdxt\" (UniqueName: \"kubernetes.io/projected/27c80de0-eb10-4c9f-ab0b-3616f87726de-kube-api-access-7jdxt\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:14:05.885018 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:05.884979    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c80de0-eb10-4c9f-ab0b-3616f87726de-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:14:06.149943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.149865    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r_27c80de0-eb10-4c9f-ab0b-3616f87726de/main/0.log"
Sep 16 13:14:06.149943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.149906    2575 generic.go:356] "Generic (PLEG): container finished" podID="27c80de0-eb10-4c9f-ab0b-3616f87726de" containerID="089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1" exitCode=1
Sep 16 13:14:06.150129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.149954    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"
Sep 16 13:14:06.150129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.149996    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r" event={"ID":"27c80de0-eb10-4c9f-ab0b-3616f87726de","Type":"ContainerDied","Data":"089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1"}
Sep 16 13:14:06.150129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.150044    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r" event={"ID":"27c80de0-eb10-4c9f-ab0b-3616f87726de","Type":"ContainerDied","Data":"38392744031a568b2292a9f0fe1b2f492ddac93d0b7ebecadf4591ac7f33a7b4"}
Sep 16 13:14:06.150129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.150064    2575 scope.go:117] "RemoveContainer" containerID="089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1"
Sep 16 13:14:06.159494 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.159480    2575 scope.go:117] "RemoveContainer" containerID="089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1"
Sep 16 13:14:06.159745 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:14:06.159726    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1\": container with ID starting with 089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1 not found: ID does not exist" containerID="089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1"
Sep 16 13:14:06.159800 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.159757    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1"} err="failed to get container status \"089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1\": rpc error: code = NotFound desc = could not find container \"089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1\": container with ID starting with 089983857fe28b310f34eec63172d05fd6d0e3223fab8585f8e5d867d1e069f1 not found: ID does not exist"
Sep 16 13:14:06.171229 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.171206    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"]
Sep 16 13:14:06.174870 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:06.174848    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc527k7r"]
Sep 16 13:14:07.328601 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:07.328559    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c80de0-eb10-4c9f-ab0b-3616f87726de" path="/var/lib/kubelet/pods/27c80de0-eb10-4c9f-ab0b-3616f87726de/volumes"
Sep 16 13:14:44.716489 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:44.716445    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:14:46.274146 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:46.274118    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:14:51.652825 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:51.652737    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:14:57.339296 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:57.339261    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:14:57.404596 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:14:57.404560    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:15:10.778399 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:10.778360    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:15:14.390721 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.390688    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"]
Sep 16 13:15:14.391086 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.391035    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" containerName="main"
Sep 16 13:15:14.391086 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.391047    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" containerName="main"
Sep 16 13:15:14.391086 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.391064    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c80de0-eb10-4c9f-ab0b-3616f87726de" containerName="main"
Sep 16 13:15:14.391086 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.391070    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c80de0-eb10-4c9f-ab0b-3616f87726de" containerName="main"
Sep 16 13:15:14.391233 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.391128    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c80de0-eb10-4c9f-ab0b-3616f87726de" containerName="main"
Sep 16 13:15:14.391233 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.391137    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ee74ee6-9140-4c9e-a3e5-7424f4e9cb4e" containerName="main"
Sep 16 13:15:14.395491 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.395471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.399547 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.399525    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-54xmz\""
Sep 16 13:15:14.399686 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.399612    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 13:15:14.399686 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.399547    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 13:15:14.399686 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.399547    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 13:15:14.404194 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.404171    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"]
Sep 16 13:15:14.419535 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.419491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:15:14.493812 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.493788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.493925 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.493815    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjjcg\" (UniqueName: \"kubernetes.io/projected/ae963eba-6e61-431d-ae0c-9e3f0c384228-kube-api-access-fjjcg\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.493976 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.493962    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.494015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.493995    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.494058 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.494023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae963eba-6e61-431d-ae0c-9e3f0c384228-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.494113 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.494088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.594482 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.594457    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.594667 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.594502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.594667 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.594537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae963eba-6e61-431d-ae0c-9e3f0c384228-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.594667 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.594568    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.594667 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.594611    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.594667 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.594635    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjjcg\" (UniqueName: \"kubernetes.io/projected/ae963eba-6e61-431d-ae0c-9e3f0c384228-kube-api-access-fjjcg\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.595000 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.594975    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.595167 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.595133    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.595280 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.595257    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.596973 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.596950    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.597398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.597377    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae963eba-6e61-431d-ae0c-9e3f0c384228-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.599555 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.599535    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"]
Sep 16 13:15:14.603281 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.603253    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.606020 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.606001    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-nl9w5\""
Sep 16 13:15:14.606437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.606415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjjcg\" (UniqueName: \"kubernetes.io/projected/ae963eba-6e61-431d-ae0c-9e3f0c384228-kube-api-access-fjjcg\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vbdg2\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.613196 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.613172    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"]
Sep 16 13:15:14.624823 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.624766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:15:14.707287 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.707224    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:14.796962 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.796932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c38f67-678d-4026-a5e1-3367cd25a315-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.797117 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.796968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68c2g\" (UniqueName: \"kubernetes.io/projected/e6c38f67-678d-4026-a5e1-3367cd25a315-kube-api-access-68c2g\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.797117 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.797076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.797117 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.797112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.897801 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.897771    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c38f67-678d-4026-a5e1-3367cd25a315-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.897926 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.897804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68c2g\" (UniqueName: \"kubernetes.io/projected/e6c38f67-678d-4026-a5e1-3367cd25a315-kube-api-access-68c2g\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.897926 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.897881    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.897926 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.897909    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.898265 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.898243    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.898355 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.898284    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.900419 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.900400    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c38f67-678d-4026-a5e1-3367cd25a315-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.906540 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.906519    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68c2g\" (UniqueName: \"kubernetes.io/projected/e6c38f67-678d-4026-a5e1-3367cd25a315-kube-api-access-68c2g\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.930405 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.930376    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:14.981660 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:14.981557    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"]
Sep 16 13:15:14.984006 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:15:14.983976    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae963eba_6e61_431d_ae0c_9e3f0c384228.slice/crio-d89b09fcac4c7b8c2cbd1f9a590110842f3fd96479203effdb9be68fad3dd994 WatchSource:0}: Error finding container d89b09fcac4c7b8c2cbd1f9a590110842f3fd96479203effdb9be68fad3dd994: Status 404 returned error can't find the container with id d89b09fcac4c7b8c2cbd1f9a590110842f3fd96479203effdb9be68fad3dd994
Sep 16 13:15:15.209808 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:15.209765    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"]
Sep 16 13:15:15.211847 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:15:15.211821    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c38f67_678d_4026_a5e1_3367cd25a315.slice/crio-dc05a84d5d634a5a2929e178d28daea92ac7fbe712d0bbea4383afbf2c6eadda WatchSource:0}: Error finding container dc05a84d5d634a5a2929e178d28daea92ac7fbe712d0bbea4383afbf2c6eadda: Status 404 returned error can't find the container with id dc05a84d5d634a5a2929e178d28daea92ac7fbe712d0bbea4383afbf2c6eadda
Sep 16 13:15:15.499227 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:15.499139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" event={"ID":"e6c38f67-678d-4026-a5e1-3367cd25a315","Type":"ContainerStarted","Data":"41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1"}
Sep 16 13:15:15.499227 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:15.499200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" event={"ID":"e6c38f67-678d-4026-a5e1-3367cd25a315","Type":"ContainerStarted","Data":"dc05a84d5d634a5a2929e178d28daea92ac7fbe712d0bbea4383afbf2c6eadda"}
Sep 16 13:15:15.499664 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:15.499228    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:15.500904 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:15.500878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2" event={"ID":"ae963eba-6e61-431d-ae0c-9e3f0c384228","Type":"ContainerStarted","Data":"f4960cbacde4d271fe439d35583d31cc443d1ef8a43cfa468b84ee400208c30a"}
Sep 16 13:15:15.501006 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:15.500924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2" event={"ID":"ae963eba-6e61-431d-ae0c-9e3f0c384228","Type":"ContainerStarted","Data":"d89b09fcac4c7b8c2cbd1f9a590110842f3fd96479203effdb9be68fad3dd994"}
Sep 16 13:15:15.517920 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:15.517874    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" podStartSLOduration=1.517860883 podStartE2EDuration="1.517860883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:15.516077752 +0000 UTC m=+1636.697220405" watchObservedRunningTime="2026-09-16 13:15:15.517860883 +0000 UTC m=+1636.699003535"
Sep 16 13:15:15.534080 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:15.534042    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2" podStartSLOduration=1.534027987 podStartE2EDuration="1.534027987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:15.532784708 +0000 UTC m=+1636.713927362" watchObservedRunningTime="2026-09-16 13:15:15.534027987 +0000 UTC m=+1636.715170640"
Sep 16 13:15:15.708017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:15.707982    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:24.708310 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:24.708275    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:24.720506 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:24.720484    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:25.530922 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:25.530886    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:15:25.564723 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:25.564692    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:34.946409 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:34.946378    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:15:35.584527 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:35.584486    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:15:46.508110 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:46.508077    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:15:48.102515 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.102472    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:15:48.111906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.111877    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"]
Sep 16 13:15:48.114322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.114297    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"]
Sep 16 13:15:48.122514 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:15:48.122479    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:15:48.122514 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:15:48.122514 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:15:48.122514 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="f4960cbacde4d271fe439d35583d31cc443d1ef8a43cfa468b84ee400208c30a" cmd=["/bin/sleep","15"]
Sep 16 13:15:48.122803 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.122536    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2" message="" err=<
Sep 16 13:15:48.122803 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:15:48.122803 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:15:48.122803 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:15:48.122803 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:15:48.122552    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:15:48.122803 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:15:48.122803 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:15:48.122803 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2" podUID="ae963eba-6e61-431d-ae0c-9e3f0c384228" containerName="main" containerID="cri-o://f4960cbacde4d271fe439d35583d31cc443d1ef8a43cfa468b84ee400208c30a"
Sep 16 13:15:48.122803 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.122620    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2" podUID="ae963eba-6e61-431d-ae0c-9e3f0c384228" containerName="main" containerID="cri-o://f4960cbacde4d271fe439d35583d31cc443d1ef8a43cfa468b84ee400208c30a" gracePeriod=60
Sep 16 13:15:48.204136 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.204094    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:15:48.554491 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.554446    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2" podUID="ae963eba-6e61-431d-ae0c-9e3f0c384228" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 16 13:15:48.672710 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.672678    2575 generic.go:356] "Generic (PLEG): container finished" podID="ae963eba-6e61-431d-ae0c-9e3f0c384228" containerID="f4960cbacde4d271fe439d35583d31cc443d1ef8a43cfa468b84ee400208c30a" exitCode=0
Sep 16 13:15:48.672887 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.672750    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2" event={"ID":"ae963eba-6e61-431d-ae0c-9e3f0c384228","Type":"ContainerDied","Data":"f4960cbacde4d271fe439d35583d31cc443d1ef8a43cfa468b84ee400208c30a"}
Sep 16 13:15:48.883441 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.883418    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:48.975990 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.975958    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjjcg\" (UniqueName: \"kubernetes.io/projected/ae963eba-6e61-431d-ae0c-9e3f0c384228-kube-api-access-fjjcg\") pod \"ae963eba-6e61-431d-ae0c-9e3f0c384228\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") "
Sep 16 13:15:48.976141 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.976003    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae963eba-6e61-431d-ae0c-9e3f0c384228-tls-certs\") pod \"ae963eba-6e61-431d-ae0c-9e3f0c384228\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") "
Sep 16 13:15:48.976141 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.976049    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-dshm\") pod \"ae963eba-6e61-431d-ae0c-9e3f0c384228\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") "
Sep 16 13:15:48.976141 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.976081    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-model-cache\") pod \"ae963eba-6e61-431d-ae0c-9e3f0c384228\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") "
Sep 16 13:15:48.976141 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.976105    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-home\") pod \"ae963eba-6e61-431d-ae0c-9e3f0c384228\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") "
Sep 16 13:15:48.976141 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.976135    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-tmp-dir\") pod \"ae963eba-6e61-431d-ae0c-9e3f0c384228\" (UID: \"ae963eba-6e61-431d-ae0c-9e3f0c384228\") "
Sep 16 13:15:48.976410 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.976334    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-home" (OuterVolumeSpecName: "home") pod "ae963eba-6e61-431d-ae0c-9e3f0c384228" (UID: "ae963eba-6e61-431d-ae0c-9e3f0c384228"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:48.976410 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.976367    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae963eba-6e61-431d-ae0c-9e3f0c384228" (UID: "ae963eba-6e61-431d-ae0c-9e3f0c384228"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:48.976521 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.976462    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae963eba-6e61-431d-ae0c-9e3f0c384228" (UID: "ae963eba-6e61-431d-ae0c-9e3f0c384228"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:48.978359 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.978330    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae963eba-6e61-431d-ae0c-9e3f0c384228-kube-api-access-fjjcg" (OuterVolumeSpecName: "kube-api-access-fjjcg") pod "ae963eba-6e61-431d-ae0c-9e3f0c384228" (UID: "ae963eba-6e61-431d-ae0c-9e3f0c384228"). InnerVolumeSpecName "kube-api-access-fjjcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:15:48.978359 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.978352    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-dshm" (OuterVolumeSpecName: "dshm") pod "ae963eba-6e61-431d-ae0c-9e3f0c384228" (UID: "ae963eba-6e61-431d-ae0c-9e3f0c384228"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:15:48.978507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:48.978377    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae963eba-6e61-431d-ae0c-9e3f0c384228-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae963eba-6e61-431d-ae0c-9e3f0c384228" (UID: "ae963eba-6e61-431d-ae0c-9e3f0c384228"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:15:49.077310 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.077286    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjjcg\" (UniqueName: \"kubernetes.io/projected/ae963eba-6e61-431d-ae0c-9e3f0c384228-kube-api-access-fjjcg\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:15:49.077310 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.077308    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae963eba-6e61-431d-ae0c-9e3f0c384228-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:15:49.077446 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.077318    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:15:49.077446 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.077326    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:15:49.077446 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.077334    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:15:49.077446 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.077341    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae963eba-6e61-431d-ae0c-9e3f0c384228-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:15:49.679492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.679465    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"
Sep 16 13:15:49.679944 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.679491    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2" event={"ID":"ae963eba-6e61-431d-ae0c-9e3f0c384228","Type":"ContainerDied","Data":"d89b09fcac4c7b8c2cbd1f9a590110842f3fd96479203effdb9be68fad3dd994"}
Sep 16 13:15:49.679944 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.679543    2575 scope.go:117] "RemoveContainer" containerID="f4960cbacde4d271fe439d35583d31cc443d1ef8a43cfa468b84ee400208c30a"
Sep 16 13:15:49.696343 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.696322    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"]
Sep 16 13:15:49.699965 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:49.699947    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vbdg2"]
Sep 16 13:15:51.329132 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:51.329094    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae963eba-6e61-431d-ae0c-9e3f0c384228" path="/var/lib/kubelet/pods/ae963eba-6e61-431d-ae0c-9e3f0c384228/volumes"
Sep 16 13:15:56.507488 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:15:56.507447    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" podUID="e6c38f67-678d-4026-a5e1-3367cd25a315" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:16:03.136085 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:03.136024    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" podUID="e6c38f67-678d-4026-a5e1-3367cd25a315" containerName="main" containerID="cri-o://41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1" gracePeriod=45
Sep 16 13:16:06.507501 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:06.507462    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" podUID="e6c38f67-678d-4026-a5e1-3367cd25a315" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:16:16.508233 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:16.508189    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" podUID="e6c38f67-678d-4026-a5e1-3367cd25a315" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:16:16.508657 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:16.508298    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:16:25.071115 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:25.071038    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:25.175466 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:25.175426    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:25.275581 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:25.275555    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:26.507494 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:26.507456    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" podUID="e6c38f67-678d-4026-a5e1-3367cd25a315" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:16:31.011496 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:31.011459    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:31.817969 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:31.817932    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:33.381343 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.381320    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:16:33.512365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.512297    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-tokenizer-cache\") pod \"e6c38f67-678d-4026-a5e1-3367cd25a315\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") "
Sep 16 13:16:33.512365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.512334    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-cachi2-cache\") pod \"e6c38f67-678d-4026-a5e1-3367cd25a315\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") "
Sep 16 13:16:33.512540 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.512375    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c38f67-678d-4026-a5e1-3367cd25a315-tls-certs\") pod \"e6c38f67-678d-4026-a5e1-3367cd25a315\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") "
Sep 16 13:16:33.512540 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.512423    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68c2g\" (UniqueName: \"kubernetes.io/projected/e6c38f67-678d-4026-a5e1-3367cd25a315-kube-api-access-68c2g\") pod \"e6c38f67-678d-4026-a5e1-3367cd25a315\" (UID: \"e6c38f67-678d-4026-a5e1-3367cd25a315\") "
Sep 16 13:16:33.512607 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.512583    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "e6c38f67-678d-4026-a5e1-3367cd25a315" (UID: "e6c38f67-678d-4026-a5e1-3367cd25a315"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:16:33.512639 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.512599    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "e6c38f67-678d-4026-a5e1-3367cd25a315" (UID: "e6c38f67-678d-4026-a5e1-3367cd25a315"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:16:33.512751 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.512734    2575 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-tokenizer-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:16:33.512796 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.512757    2575 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c38f67-678d-4026-a5e1-3367cd25a315-cachi2-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:16:33.514596 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.514575    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c38f67-678d-4026-a5e1-3367cd25a315-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6c38f67-678d-4026-a5e1-3367cd25a315" (UID: "e6c38f67-678d-4026-a5e1-3367cd25a315"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:16:33.514698 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.514680    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c38f67-678d-4026-a5e1-3367cd25a315-kube-api-access-68c2g" (OuterVolumeSpecName: "kube-api-access-68c2g") pod "e6c38f67-678d-4026-a5e1-3367cd25a315" (UID: "e6c38f67-678d-4026-a5e1-3367cd25a315"). InnerVolumeSpecName "kube-api-access-68c2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:33.613615 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.613584    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c38f67-678d-4026-a5e1-3367cd25a315-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:16:33.613615 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.613609    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68c2g\" (UniqueName: \"kubernetes.io/projected/e6c38f67-678d-4026-a5e1-3367cd25a315-kube-api-access-68c2g\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:16:33.909598 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.909562    2575 generic.go:356] "Generic (PLEG): container finished" podID="e6c38f67-678d-4026-a5e1-3367cd25a315" containerID="41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1" exitCode=0
Sep 16 13:16:33.909816 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.909605    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"
Sep 16 13:16:33.909816 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.909629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" event={"ID":"e6c38f67-678d-4026-a5e1-3367cd25a315","Type":"ContainerDied","Data":"41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1"}
Sep 16 13:16:33.909816 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.909713    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv" event={"ID":"e6c38f67-678d-4026-a5e1-3367cd25a315","Type":"ContainerDied","Data":"dc05a84d5d634a5a2929e178d28daea92ac7fbe712d0bbea4383afbf2c6eadda"}
Sep 16 13:16:33.909816 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.909746    2575 scope.go:117] "RemoveContainer" containerID="41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1"
Sep 16 13:16:33.918841 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.918825    2575 scope.go:117] "RemoveContainer" containerID="41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1"
Sep 16 13:16:33.919102 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:16:33.919081    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1\": container with ID starting with 41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1 not found: ID does not exist" containerID="41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1"
Sep 16 13:16:33.919194 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.919107    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1"} err="failed to get container status \"41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1\": rpc error: code = NotFound desc = could not find container \"41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1\": container with ID starting with 41dd9827c976989fdbfd2777d60600ea23c4a9be4b16371173fc6bcd89795db1 not found: ID does not exist"
Sep 16 13:16:33.932578 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.932554    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"]
Sep 16 13:16:33.934871 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:33.934853    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcjj9kv"]
Sep 16 13:16:35.328939 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:35.328911    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c38f67-678d-4026-a5e1-3367cd25a315" path="/var/lib/kubelet/pods/e6c38f67-678d-4026-a5e1-3367cd25a315/volumes"
Sep 16 13:16:37.468467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:37.468436    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:37.522691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:37.522639    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:37.890507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:37.890469    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:37.989019 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:37.988990    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:44.445699 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:44.445627    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:46.372839 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.372804    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"]
Sep 16 13:16:46.373355 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.373276    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c38f67-678d-4026-a5e1-3367cd25a315" containerName="main"
Sep 16 13:16:46.373355 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.373294    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c38f67-678d-4026-a5e1-3367cd25a315" containerName="main"
Sep 16 13:16:46.373355 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.373325    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae963eba-6e61-431d-ae0c-9e3f0c384228" containerName="main"
Sep 16 13:16:46.373355 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.373336    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae963eba-6e61-431d-ae0c-9e3f0c384228" containerName="main"
Sep 16 13:16:46.373578 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.373418    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c38f67-678d-4026-a5e1-3367cd25a315" containerName="main"
Sep 16 13:16:46.373578 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.373431    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae963eba-6e61-431d-ae0c-9e3f0c384228" containerName="main"
Sep 16 13:16:46.376495 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.376480    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.380331 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.380300    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 16 13:16:46.380463 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.380301    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 13:16:46.380463 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.380347    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 13:16:46.380463 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.380357    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-rxzlt\""
Sep 16 13:16:46.385501 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.385480    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"]
Sep 16 13:16:46.411698 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.411672    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-dshm\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.411801 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.411711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhp4\" (UniqueName: \"kubernetes.io/projected/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-kube-api-access-7dhp4\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.411801 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.411747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.411801 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.411769    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.411934 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.411804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-home\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.411934 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.411827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.488639 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.488612    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:46.512767 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.512737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.512950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.512778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-home\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.512950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.512799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.512950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.512847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-dshm\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.512950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.512869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhp4\" (UniqueName: \"kubernetes.io/projected/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-kube-api-access-7dhp4\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.512950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.512888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.513225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.513120    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.513225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.513190    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-home\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.513303 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.513268    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.515302 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.515279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-dshm\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.515585 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.515566    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.521377 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.521351    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhp4\" (UniqueName: \"kubernetes.io/projected/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-kube-api-access-7dhp4\") pod \"canary-v2-kserve-7f65dbbccc-7lfrl\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.687690 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.687589    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:46.967760 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.967737    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"]
Sep 16 13:16:46.969684 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:16:46.969636    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a63a4e1_dc5e_429e_8a6f_9193df8d891d.slice/crio-fe6ab7d385c64dc2b64a2872b5d81979541c2d83c8305c3bb710748e4ef0efaa WatchSource:0}: Error finding container fe6ab7d385c64dc2b64a2872b5d81979541c2d83c8305c3bb710748e4ef0efaa: Status 404 returned error can't find the container with id fe6ab7d385c64dc2b64a2872b5d81979541c2d83c8305c3bb710748e4ef0efaa
Sep 16 13:16:46.971618 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.971602    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:16:46.977312 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:46.977288    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl" event={"ID":"4a63a4e1-dc5e-429e-8a6f-9193df8d891d","Type":"ContainerStarted","Data":"fe6ab7d385c64dc2b64a2872b5d81979541c2d83c8305c3bb710748e4ef0efaa"}
Sep 16 13:16:48.989218 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:48.989186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl" event={"ID":"4a63a4e1-dc5e-429e-8a6f-9193df8d891d","Type":"ContainerStarted","Data":"b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944"}
Sep 16 13:16:49.008066 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:49.008018    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl" podStartSLOduration=2.01360743 podStartE2EDuration="3.008003301s" totalImagesPullingTime="994.395871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:16:49.005751125 +0000 UTC m=+1730.186893812" watchObservedRunningTime="2026-09-16 13:16:49.008003301 +0000 UTC m=+1730.189145953"
Sep 16 13:16:49.688264 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:49.688219    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:50.150359 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:50.150322    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:50.222350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:50.222319    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:16:56.688366 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:56.688331    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:56.700465 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:56.700440    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:16:57.042747 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:16:57.042633    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:17:07.819495 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.819444    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"]
Sep 16 13:17:07.824060 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.824038    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.828329 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.828301    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 13:17:07.828329 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.828323    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-8l8zm\""
Sep 16 13:17:07.828547 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.828329    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 13:17:07.828547 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.828527    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 13:17:07.828692 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.828601    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-brmgb\""
Sep 16 13:17:07.837816 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.837797    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"]
Sep 16 13:17:07.867153 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.867125    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:17:07.884254 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.884224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.884254 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.884252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.884400 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.884286    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721c406-54cc-472c-a53a-b20dabe15234-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.884400 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.884353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.884400 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.884380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.884514 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.884419    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqmdj\" (UniqueName: \"kubernetes.io/projected/6721c406-54cc-472c-a53a-b20dabe15234-kube-api-access-pqmdj\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.985486 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.985462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721c406-54cc-472c-a53a-b20dabe15234-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.985591 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.985496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.985591 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.985513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.985591 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.985543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqmdj\" (UniqueName: \"kubernetes.io/projected/6721c406-54cc-472c-a53a-b20dabe15234-kube-api-access-pqmdj\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.985780 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.985605    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.985780 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.985630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.985978 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.985956    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.986107 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.985987    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.986107 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.986042    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.987993 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.987977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.988149 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.988130    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721c406-54cc-472c-a53a-b20dabe15234-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:07.993911 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:07.993888    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqmdj\" (UniqueName: \"kubernetes.io/projected/6721c406-54cc-472c-a53a-b20dabe15234-kube-api-access-pqmdj\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:08.080863 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:08.080786    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:17:08.136095 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:08.136068    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:08.417829 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:08.417804    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"]
Sep 16 13:17:08.420410 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:17:08.420386    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6721c406_54cc_472c_a53a_b20dabe15234.slice/crio-d9fc61f902b13bd9c202bfc9f3054ad076cacc6d7a5b56424d575e3990d604dc WatchSource:0}: Error finding container d9fc61f902b13bd9c202bfc9f3054ad076cacc6d7a5b56424d575e3990d604dc: Status 404 returned error can't find the container with id d9fc61f902b13bd9c202bfc9f3054ad076cacc6d7a5b56424d575e3990d604dc
Sep 16 13:17:09.100295 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:09.100262    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" event={"ID":"6721c406-54cc-472c-a53a-b20dabe15234","Type":"ContainerStarted","Data":"d9fc61f902b13bd9c202bfc9f3054ad076cacc6d7a5b56424d575e3990d604dc"}
Sep 16 13:17:11.112408 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:11.112367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" event={"ID":"6721c406-54cc-472c-a53a-b20dabe15234","Type":"ContainerStarted","Data":"167393e15524dc679bc40d20722b59cea9c9bcb7e1b19f9fbaf042fc4428a3b6"}
Sep 16 13:17:11.112889 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:11.112512    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:12.119883 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:12.119850    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" event={"ID":"6721c406-54cc-472c-a53a-b20dabe15234","Type":"ContainerStarted","Data":"311381353e08d679dd8c20b185c8b4de302b25e226b3b7fde71564e55dc104f5"}
Sep 16 13:17:12.136259 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:12.136234    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:12.140132 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:12.140090    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" podStartSLOduration=3.439552154 podStartE2EDuration="5.140072885s" totalImagesPullingTime="1.700520731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:17:12.139367058 +0000 UTC m=+1753.320509712" watchObservedRunningTime="2026-09-16 13:17:12.140072885 +0000 UTC m=+1753.321215578"
Sep 16 13:17:18.137188 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:18.137155    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:18.149577 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:18.149551    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:18.165256 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:18.165229    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:29.172478 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:29.172447    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:17:34.339423 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:17:34.339387    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:18:09.305823 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:09.305787    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:18:38.123937 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:38.123901    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:18:38.257508 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:38.257477    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:18:38.390971 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:38.390883    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:18:40.297717 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:40.297681    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:18:43.095461 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:43.095428    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"]
Sep 16 13:18:43.190899 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:43.190867    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:18:57.057716 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.057680    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"]
Sep 16 13:18:57.061272 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.061255    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.069188 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.069163    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 13:18:57.069347 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.069234    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-bfd5s\""
Sep 16 13:18:57.069347 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.069275    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 13:18:57.069518 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.069288    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 13:18:57.081401 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.081375    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"]
Sep 16 13:18:57.097995 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.097967    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:18:57.167304 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.167273    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2jj\" (UniqueName: \"kubernetes.io/projected/e0407135-c489-4689-869f-ce22d4e5e1e9-kube-api-access-fp2jj\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.167443 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.167326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-model-cache\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.167443 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.167363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0407135-c489-4689-869f-ce22d4e5e1e9-tls-certs\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.167443 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.167393    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-home\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.167553 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.167462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.167553 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.167493    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-dshm\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.268267 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.268230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-dshm\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.268267 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.268269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2jj\" (UniqueName: \"kubernetes.io/projected/e0407135-c489-4689-869f-ce22d4e5e1e9-kube-api-access-fp2jj\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.268490 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.268330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-model-cache\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.268490 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.268368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0407135-c489-4689-869f-ce22d4e5e1e9-tls-certs\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.268605 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.268508    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-home\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.268605 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.268584    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.268844 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.268820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-model-cache\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.268844 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.268840    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-home\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.268946 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.268924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.270898 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.270875    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-dshm\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.271137 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.271123    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0407135-c489-4689-869f-ce22d4e5e1e9-tls-certs\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.279952 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.279930    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2jj\" (UniqueName: \"kubernetes.io/projected/e0407135-c489-4689-869f-ce22d4e5e1e9-kube-api-access-fp2jj\") pod \"test-div-v1-kserve-fb566d667-8mb8p\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.372732 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.372701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:18:57.680463 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.680424    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"]
Sep 16 13:18:57.681123 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:18:57.681096    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0407135_c489_4689_869f_ce22d4e5e1e9.slice/crio-0617a7a76494c344f8d26dda92ea6a3359114828a30f898aa441ac6302ff024d WatchSource:0}: Error finding container 0617a7a76494c344f8d26dda92ea6a3359114828a30f898aa441ac6302ff024d: Status 404 returned error can't find the container with id 0617a7a76494c344f8d26dda92ea6a3359114828a30f898aa441ac6302ff024d
Sep 16 13:18:57.693497 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:57.693471    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p" event={"ID":"e0407135-c489-4689-869f-ce22d4e5e1e9","Type":"ContainerStarted","Data":"0617a7a76494c344f8d26dda92ea6a3359114828a30f898aa441ac6302ff024d"}
Sep 16 13:18:58.115332 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.115296    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl" podUID="4a63a4e1-dc5e-429e-8a6f-9193df8d891d" containerName="main" containerID="cri-o://b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944" gracePeriod=45
Sep 16 13:18:58.362550 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.362528    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:18:58.377456 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.377414    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dhp4\" (UniqueName: \"kubernetes.io/projected/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-kube-api-access-7dhp4\") pod \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") "
Sep 16 13:18:58.377456 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.377455    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tls-certs\") pod \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") "
Sep 16 13:18:58.377570 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.377508    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-model-cache\") pod \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") "
Sep 16 13:18:58.377570 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.377550    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tmp-dir\") pod \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") "
Sep 16 13:18:58.377710 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.377588    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-dshm\") pod \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") "
Sep 16 13:18:58.377710 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.377610    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-home\") pod \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\" (UID: \"4a63a4e1-dc5e-429e-8a6f-9193df8d891d\") "
Sep 16 13:18:58.377866 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.377818    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a63a4e1-dc5e-429e-8a6f-9193df8d891d" (UID: "4a63a4e1-dc5e-429e-8a6f-9193df8d891d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:18:58.377980 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.377870    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a63a4e1-dc5e-429e-8a6f-9193df8d891d" (UID: "4a63a4e1-dc5e-429e-8a6f-9193df8d891d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:18:58.377980 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.377965    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-home" (OuterVolumeSpecName: "home") pod "4a63a4e1-dc5e-429e-8a6f-9193df8d891d" (UID: "4a63a4e1-dc5e-429e-8a6f-9193df8d891d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:18:58.379857 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.379836    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-dshm" (OuterVolumeSpecName: "dshm") pod "4a63a4e1-dc5e-429e-8a6f-9193df8d891d" (UID: "4a63a4e1-dc5e-429e-8a6f-9193df8d891d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:18:58.379966 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.379926    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-kube-api-access-7dhp4" (OuterVolumeSpecName: "kube-api-access-7dhp4") pod "4a63a4e1-dc5e-429e-8a6f-9193df8d891d" (UID: "4a63a4e1-dc5e-429e-8a6f-9193df8d891d"). InnerVolumeSpecName "kube-api-access-7dhp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:18:58.379966 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.379932    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a63a4e1-dc5e-429e-8a6f-9193df8d891d" (UID: "4a63a4e1-dc5e-429e-8a6f-9193df8d891d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:18:58.478578 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.478547    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:18:58.478578 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.478572    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:18:58.478578 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.478581    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:18:58.478822 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.478589    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:18:58.478822 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.478598    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhp4\" (UniqueName: \"kubernetes.io/projected/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-kube-api-access-7dhp4\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:18:58.478822 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.478606    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a63a4e1-dc5e-429e-8a6f-9193df8d891d-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:18:58.705705 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.705595    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p" event={"ID":"e0407135-c489-4689-869f-ce22d4e5e1e9","Type":"ContainerStarted","Data":"cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4"}
Sep 16 13:18:58.706999 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.706971    2575 generic.go:356] "Generic (PLEG): container finished" podID="4a63a4e1-dc5e-429e-8a6f-9193df8d891d" containerID="b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944" exitCode=0
Sep 16 13:18:58.707189 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.707016    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"
Sep 16 13:18:58.707189 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.707055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl" event={"ID":"4a63a4e1-dc5e-429e-8a6f-9193df8d891d","Type":"ContainerDied","Data":"b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944"}
Sep 16 13:18:58.707189 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.707112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl" event={"ID":"4a63a4e1-dc5e-429e-8a6f-9193df8d891d","Type":"ContainerDied","Data":"fe6ab7d385c64dc2b64a2872b5d81979541c2d83c8305c3bb710748e4ef0efaa"}
Sep 16 13:18:58.707189 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.707146    2575 scope.go:117] "RemoveContainer" containerID="b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944"
Sep 16 13:18:58.716287 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.716271    2575 scope.go:117] "RemoveContainer" containerID="b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944"
Sep 16 13:18:58.716555 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:18:58.716539    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944\": container with ID starting with b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944 not found: ID does not exist" containerID="b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944"
Sep 16 13:18:58.716608 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.716562    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944"} err="failed to get container status \"b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944\": rpc error: code = NotFound desc = could not find container \"b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944\": container with ID starting with b27d07cf50a3167a3adda368956c8808913f42b862ea87e93f0700601407a944 not found: ID does not exist"
Sep 16 13:18:58.775247 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.775204    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p" podStartSLOduration=1.775191023 podStartE2EDuration="1.775191023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:18:58.749894312 +0000 UTC m=+1859.931036965" watchObservedRunningTime="2026-09-16 13:18:58.775191023 +0000 UTC m=+1859.956333675"
Sep 16 13:18:58.777047 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.777031    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"]
Sep 16 13:18:58.786090 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:58.786067    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-7lfrl"]
Sep 16 13:18:59.056371 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:59.056271    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:18:59.331625 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:59.331547    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a63a4e1-dc5e-429e-8a6f-9193df8d891d" path="/var/lib/kubelet/pods/4a63a4e1-dc5e-429e-8a6f-9193df8d891d/volumes"
Sep 16 13:18:59.373530 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:18:59.373494    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:19:04.293965 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:04.293929    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:04.395790 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:04.395762    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:07.373342 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:07.373308    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:19:07.385720 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:07.385689    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:19:07.764157 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:07.764076    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:19:16.401616 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.401581    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"]
Sep 16 13:19:16.402131 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.401974    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a63a4e1-dc5e-429e-8a6f-9193df8d891d" containerName="main"
Sep 16 13:19:16.402131 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.401986    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a63a4e1-dc5e-429e-8a6f-9193df8d891d" containerName="main"
Sep 16 13:19:16.402131 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.402046    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a63a4e1-dc5e-429e-8a6f-9193df8d891d" containerName="main"
Sep 16 13:19:16.405156 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.405135    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.408233 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.408209    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 13:19:16.421970 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.421944    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"]
Sep 16 13:19:16.462532 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.462502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:16.522787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.522762    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc627\" (UniqueName: \"kubernetes.io/projected/031b78a8-d805-410d-a0b0-f938786ee383-kube-api-access-sc627\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.522917 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.522796    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.522917 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.522812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b78a8-d805-410d-a0b0-f938786ee383-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.522917 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.522831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.523044 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.522965    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.523044 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.523008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-home\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.624068 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.624040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.624214 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.624091    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-home\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.624214 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.624125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc627\" (UniqueName: \"kubernetes.io/projected/031b78a8-d805-410d-a0b0-f938786ee383-kube-api-access-sc627\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.624214 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.624154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.624214 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.624176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b78a8-d805-410d-a0b0-f938786ee383-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.624214 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.624204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.624562 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.624538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.624823 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.624772    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.624823 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.624685    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-home\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.626635 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.626611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.627229 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.627213    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b78a8-d805-410d-a0b0-f938786ee383-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.633912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.633891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc627\" (UniqueName: \"kubernetes.io/projected/031b78a8-d805-410d-a0b0-f938786ee383-kube-api-access-sc627\") pod \"test-div-v2-kserve-5f6fdb6b77-kmx46\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:16.716213 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:16.716140    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:17.000616 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:17.000514    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"]
Sep 16 13:19:17.003334 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:19:17.003305    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031b78a8_d805_410d_a0b0_f938786ee383.slice/crio-849328be5fdcd69143a28a444d7aa5fac3fb39aced502fc76a48b40dadb34bc5 WatchSource:0}: Error finding container 849328be5fdcd69143a28a444d7aa5fac3fb39aced502fc76a48b40dadb34bc5: Status 404 returned error can't find the container with id 849328be5fdcd69143a28a444d7aa5fac3fb39aced502fc76a48b40dadb34bc5
Sep 16 13:19:17.808137 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:17.808103    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46" event={"ID":"031b78a8-d805-410d-a0b0-f938786ee383","Type":"ContainerStarted","Data":"a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd"}
Sep 16 13:19:17.808545 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:17.808160    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46" event={"ID":"031b78a8-d805-410d-a0b0-f938786ee383","Type":"ContainerStarted","Data":"849328be5fdcd69143a28a444d7aa5fac3fb39aced502fc76a48b40dadb34bc5"}
Sep 16 13:19:17.835299 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:17.835249    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46" podStartSLOduration=1.8352349989999999 podStartE2EDuration="1.835234999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:19:17.833610719 +0000 UTC m=+1879.014753384" watchObservedRunningTime="2026-09-16 13:19:17.835234999 +0000 UTC m=+1879.016377653"
Sep 16 13:19:18.716928 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:18.716885    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:18.961680 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:18.961615    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:18.994433 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:18.994363    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"]
Sep 16 13:19:19.003819 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:19:19.003785    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:19:19.003819 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:19:19.003819 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:19:19.003819 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="311381353e08d679dd8c20b185c8b4de302b25e226b3b7fde71564e55dc104f5" cmd=["/bin/sleep","15"]
Sep 16 13:19:19.004042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.003832    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" message="" err=<
Sep 16 13:19:19.004042 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:19:19.004042 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:19:19.004042 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:19:19.004042 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:19:19.003844    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:19:19.004042 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:19:19.004042 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:19:19.004042 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="main" containerID="cri-o://311381353e08d679dd8c20b185c8b4de302b25e226b3b7fde71564e55dc104f5"
Sep 16 13:19:19.004042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.003875    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="main" containerID="cri-o://311381353e08d679dd8c20b185c8b4de302b25e226b3b7fde71564e55dc104f5" gracePeriod=60
Sep 16 13:19:19.089128 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.089074    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:19.154767 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.154737    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 16 13:19:19.617365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.617336    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="llm-d-routing-sidecar" containerID="cri-o://167393e15524dc679bc40d20722b59cea9c9bcb7e1b19f9fbaf042fc4428a3b6" gracePeriod=60
Sep 16 13:19:19.823272 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.823236    2575 generic.go:356] "Generic (PLEG): container finished" podID="6721c406-54cc-472c-a53a-b20dabe15234" containerID="311381353e08d679dd8c20b185c8b4de302b25e226b3b7fde71564e55dc104f5" exitCode=0
Sep 16 13:19:19.823272 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.823268    2575 generic.go:356] "Generic (PLEG): container finished" podID="6721c406-54cc-472c-a53a-b20dabe15234" containerID="167393e15524dc679bc40d20722b59cea9c9bcb7e1b19f9fbaf042fc4428a3b6" exitCode=0
Sep 16 13:19:19.823499 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.823303    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" event={"ID":"6721c406-54cc-472c-a53a-b20dabe15234","Type":"ContainerDied","Data":"311381353e08d679dd8c20b185c8b4de302b25e226b3b7fde71564e55dc104f5"}
Sep 16 13:19:19.823499 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.823361    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" event={"ID":"6721c406-54cc-472c-a53a-b20dabe15234","Type":"ContainerDied","Data":"167393e15524dc679bc40d20722b59cea9c9bcb7e1b19f9fbaf042fc4428a3b6"}
Sep 16 13:19:19.874694 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.874612    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:19:19.953576 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.953549    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-home\") pod \"6721c406-54cc-472c-a53a-b20dabe15234\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") "
Sep 16 13:19:19.953781 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.953585    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-model-cache\") pod \"6721c406-54cc-472c-a53a-b20dabe15234\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") "
Sep 16 13:19:19.953781 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.953621    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-dshm\") pod \"6721c406-54cc-472c-a53a-b20dabe15234\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") "
Sep 16 13:19:19.953781 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.953657    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-tmp-dir\") pod \"6721c406-54cc-472c-a53a-b20dabe15234\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") "
Sep 16 13:19:19.953781 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.953742    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721c406-54cc-472c-a53a-b20dabe15234-tls-certs\") pod \"6721c406-54cc-472c-a53a-b20dabe15234\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") "
Sep 16 13:19:19.953995 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.953796    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqmdj\" (UniqueName: \"kubernetes.io/projected/6721c406-54cc-472c-a53a-b20dabe15234-kube-api-access-pqmdj\") pod \"6721c406-54cc-472c-a53a-b20dabe15234\" (UID: \"6721c406-54cc-472c-a53a-b20dabe15234\") "
Sep 16 13:19:19.953995 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.953910    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-home" (OuterVolumeSpecName: "home") pod "6721c406-54cc-472c-a53a-b20dabe15234" (UID: "6721c406-54cc-472c-a53a-b20dabe15234"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:19.953995 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.953914    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-model-cache" (OuterVolumeSpecName: "model-cache") pod "6721c406-54cc-472c-a53a-b20dabe15234" (UID: "6721c406-54cc-472c-a53a-b20dabe15234"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:19.953995 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.953933    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6721c406-54cc-472c-a53a-b20dabe15234" (UID: "6721c406-54cc-472c-a53a-b20dabe15234"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:19.954190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.954093    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:19.954190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.954111    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:19.954289 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.954203    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:19.956001 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.955977    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-dshm" (OuterVolumeSpecName: "dshm") pod "6721c406-54cc-472c-a53a-b20dabe15234" (UID: "6721c406-54cc-472c-a53a-b20dabe15234"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:19.956450 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.956432    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6721c406-54cc-472c-a53a-b20dabe15234-kube-api-access-pqmdj" (OuterVolumeSpecName: "kube-api-access-pqmdj") pod "6721c406-54cc-472c-a53a-b20dabe15234" (UID: "6721c406-54cc-472c-a53a-b20dabe15234"). InnerVolumeSpecName "kube-api-access-pqmdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:19:19.956535 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:19.956428    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6721c406-54cc-472c-a53a-b20dabe15234-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6721c406-54cc-472c-a53a-b20dabe15234" (UID: "6721c406-54cc-472c-a53a-b20dabe15234"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:19:20.055290 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:20.055261    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6721c406-54cc-472c-a53a-b20dabe15234-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:20.055290 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:20.055286    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqmdj\" (UniqueName: \"kubernetes.io/projected/6721c406-54cc-472c-a53a-b20dabe15234-kube-api-access-pqmdj\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:20.055290 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:20.055296    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6721c406-54cc-472c-a53a-b20dabe15234-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:20.830366 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:20.830285    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"
Sep 16 13:19:20.830366 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:20.830301    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq" event={"ID":"6721c406-54cc-472c-a53a-b20dabe15234","Type":"ContainerDied","Data":"d9fc61f902b13bd9c202bfc9f3054ad076cacc6d7a5b56424d575e3990d604dc"}
Sep 16 13:19:20.830366 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:20.830362    2575 scope.go:117] "RemoveContainer" containerID="311381353e08d679dd8c20b185c8b4de302b25e226b3b7fde71564e55dc104f5"
Sep 16 13:19:20.840034 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:20.840015    2575 scope.go:117] "RemoveContainer" containerID="167393e15524dc679bc40d20722b59cea9c9bcb7e1b19f9fbaf042fc4428a3b6"
Sep 16 13:19:20.862355 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:20.862329    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"]
Sep 16 13:19:20.866351 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:20.866332    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-d7dqq"]
Sep 16 13:19:21.331375 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:21.331344    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6721c406-54cc-472c-a53a-b20dabe15234" path="/var/lib/kubelet/pods/6721c406-54cc-472c-a53a-b20dabe15234/volumes"
Sep 16 13:19:21.766902 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:21.766864    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:26.716698 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:26.716640    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:26.729270 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:26.729246    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:26.872279 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:26.872249    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:19:28.347743 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:28.347715    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:28.454906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:28.454880    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:39.942231 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:39.942185    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:40.062601 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:40.062575    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:44.266375 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:44.266344    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:44.306140 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:44.306108    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"]
Sep 16 13:19:44.375102 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:44.375072    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:44.466483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:44.466448    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:45.155748 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:45.155711    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"]
Sep 16 13:19:45.197691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:45.197658    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:49.551350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:49.551314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:52.312249 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.312214    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"]
Sep 16 13:19:52.312630 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.312596    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="llm-d-routing-sidecar"
Sep 16 13:19:52.312630 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.312607    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="llm-d-routing-sidecar"
Sep 16 13:19:52.312630 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.312626    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="main"
Sep 16 13:19:52.312630 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.312631    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="main"
Sep 16 13:19:52.312842 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.312729    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="llm-d-routing-sidecar"
Sep 16 13:19:52.312842 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.312743    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6721c406-54cc-472c-a53a-b20dabe15234" containerName="main"
Sep 16 13:19:52.316175 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.316156    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.319794 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.319772    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 13:19:52.319916 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.319822    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 13:19:52.321017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.320996    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 13:19:52.321137 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.321120    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-fv66k\""
Sep 16 13:19:52.326521 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.326499    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"]
Sep 16 13:19:52.343243 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.343208    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:52.515810 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.515776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6030c021-13b8-44ab-a0f7-a32af63bb308-tls-certs\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.515934 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.515816    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-model-cache\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.515934 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.515846    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-dshm\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.515934 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.515869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxd2\" (UniqueName: \"kubernetes.io/projected/6030c021-13b8-44ab-a0f7-a32af63bb308-kube-api-access-7hxd2\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.515934 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.515890    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.516260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.515952    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-home\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.616315 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.616283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.616481 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.616347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-home\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.616481 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.616412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6030c021-13b8-44ab-a0f7-a32af63bb308-tls-certs\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.616481 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.616441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-model-cache\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.616481 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.616469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-dshm\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.616737 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.616493    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxd2\" (UniqueName: \"kubernetes.io/projected/6030c021-13b8-44ab-a0f7-a32af63bb308-kube-api-access-7hxd2\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.616737 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.616638    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.616849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.616736    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-home\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.616849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.616813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-model-cache\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.618785 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.618762    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-dshm\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.618954 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.618938    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6030c021-13b8-44ab-a0f7-a32af63bb308-tls-certs\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.626730 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.626706    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxd2\" (UniqueName: \"kubernetes.io/projected/6030c021-13b8-44ab-a0f7-a32af63bb308-kube-api-access-7hxd2\") pod \"test-div-v1-kserve-fb566d667-kb45f\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.628354 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.628307    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:52.913850 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.913823    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"]
Sep 16 13:19:52.915200 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:19:52.915178    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6030c021_13b8_44ab_a0f7_a32af63bb308.slice/crio-daa7ddda177936fbbfb9b09490dea42a9879276eb463ac098a85d724a31cb530 WatchSource:0}: Error finding container daa7ddda177936fbbfb9b09490dea42a9879276eb463ac098a85d724a31cb530: Status 404 returned error can't find the container with id daa7ddda177936fbbfb9b09490dea42a9879276eb463ac098a85d724a31cb530
Sep 16 13:19:52.995770 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.995745    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f" event={"ID":"6030c021-13b8-44ab-a0f7-a32af63bb308","Type":"ContainerStarted","Data":"9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6"}
Sep 16 13:19:52.995870 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:52.995796    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f" event={"ID":"6030c021-13b8-44ab-a0f7-a32af63bb308","Type":"ContainerStarted","Data":"daa7ddda177936fbbfb9b09490dea42a9879276eb463ac098a85d724a31cb530"}
Sep 16 13:19:53.024904 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:53.024863    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f" podStartSLOduration=1.024848343 podStartE2EDuration="1.024848343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:19:53.022430875 +0000 UTC m=+1914.203573531" watchObservedRunningTime="2026-09-16 13:19:53.024848343 +0000 UTC m=+1914.205990996"
Sep 16 13:19:53.629438 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:53.629390    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:19:54.606999 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:54.606961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:19:59.327114 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.327080    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p" podUID="e0407135-c489-4689-869f-ce22d4e5e1e9" containerName="main" containerID="cri-o://cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4" gracePeriod=45
Sep 16 13:19:59.583292 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.583234    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:19:59.673804 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.673780    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-model-cache\") pod \"e0407135-c489-4689-869f-ce22d4e5e1e9\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") "
Sep 16 13:19:59.673943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.673830    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-dshm\") pod \"e0407135-c489-4689-869f-ce22d4e5e1e9\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") "
Sep 16 13:19:59.673943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.673899    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0407135-c489-4689-869f-ce22d4e5e1e9-tls-certs\") pod \"e0407135-c489-4689-869f-ce22d4e5e1e9\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") "
Sep 16 13:19:59.673943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.673940    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp2jj\" (UniqueName: \"kubernetes.io/projected/e0407135-c489-4689-869f-ce22d4e5e1e9-kube-api-access-fp2jj\") pod \"e0407135-c489-4689-869f-ce22d4e5e1e9\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") "
Sep 16 13:19:59.674076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.673962    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-tmp-dir\") pod \"e0407135-c489-4689-869f-ce22d4e5e1e9\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") "
Sep 16 13:19:59.674076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.673993    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-home\") pod \"e0407135-c489-4689-869f-ce22d4e5e1e9\" (UID: \"e0407135-c489-4689-869f-ce22d4e5e1e9\") "
Sep 16 13:19:59.674076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.674068    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0407135-c489-4689-869f-ce22d4e5e1e9" (UID: "e0407135-c489-4689-869f-ce22d4e5e1e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:59.674261 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.674244    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:59.674328 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.674256    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0407135-c489-4689-869f-ce22d4e5e1e9" (UID: "e0407135-c489-4689-869f-ce22d4e5e1e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:59.674328 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.674266    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-home" (OuterVolumeSpecName: "home") pod "e0407135-c489-4689-869f-ce22d4e5e1e9" (UID: "e0407135-c489-4689-869f-ce22d4e5e1e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:59.676304 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.676282    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-dshm" (OuterVolumeSpecName: "dshm") pod "e0407135-c489-4689-869f-ce22d4e5e1e9" (UID: "e0407135-c489-4689-869f-ce22d4e5e1e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:19:59.676390 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.676300    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0407135-c489-4689-869f-ce22d4e5e1e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0407135-c489-4689-869f-ce22d4e5e1e9" (UID: "e0407135-c489-4689-869f-ce22d4e5e1e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:19:59.676390 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.676375    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0407135-c489-4689-869f-ce22d4e5e1e9-kube-api-access-fp2jj" (OuterVolumeSpecName: "kube-api-access-fp2jj") pod "e0407135-c489-4689-869f-ce22d4e5e1e9" (UID: "e0407135-c489-4689-869f-ce22d4e5e1e9"). InnerVolumeSpecName "kube-api-access-fp2jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:19:59.775076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.775052    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0407135-c489-4689-869f-ce22d4e5e1e9-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:59.775076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.775072    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2jj\" (UniqueName: \"kubernetes.io/projected/e0407135-c489-4689-869f-ce22d4e5e1e9-kube-api-access-fp2jj\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:59.775212 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.775082    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:59.775212 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.775090    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:19:59.775212 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:19:59.775098    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0407135-c489-4689-869f-ce22d4e5e1e9-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:20:00.035851 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.035821    2575 generic.go:356] "Generic (PLEG): container finished" podID="e0407135-c489-4689-869f-ce22d4e5e1e9" containerID="cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4" exitCode=0
Sep 16 13:20:00.036025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.035869    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"
Sep 16 13:20:00.036025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.035913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p" event={"ID":"e0407135-c489-4689-869f-ce22d4e5e1e9","Type":"ContainerDied","Data":"cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4"}
Sep 16 13:20:00.036025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.035973    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p" event={"ID":"e0407135-c489-4689-869f-ce22d4e5e1e9","Type":"ContainerDied","Data":"0617a7a76494c344f8d26dda92ea6a3359114828a30f898aa441ac6302ff024d"}
Sep 16 13:20:00.036025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.036005    2575 scope.go:117] "RemoveContainer" containerID="cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4"
Sep 16 13:20:00.045176 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.045157    2575 scope.go:117] "RemoveContainer" containerID="cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4"
Sep 16 13:20:00.045420 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:20:00.045399    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4\": container with ID starting with cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4 not found: ID does not exist" containerID="cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4"
Sep 16 13:20:00.045492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.045428    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4"} err="failed to get container status \"cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4\": rpc error: code = NotFound desc = could not find container \"cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4\": container with ID starting with cee548fab4842d0c23fa0c240c66718d608322ea37c9c3bb5ae04f9dd23251b4 not found: ID does not exist"
Sep 16 13:20:00.085010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.084988    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"]
Sep 16 13:20:00.096312 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.096289    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-8mb8p"]
Sep 16 13:20:00.174466 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.174439    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46" podUID="031b78a8-d805-410d-a0b0-f938786ee383" containerName="main" containerID="cri-o://a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd" gracePeriod=45
Sep 16 13:20:00.425401 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.425378    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:20:00.481544 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.481519    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-model-cache\") pod \"031b78a8-d805-410d-a0b0-f938786ee383\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") "
Sep 16 13:20:00.481720 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.481565    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-home\") pod \"031b78a8-d805-410d-a0b0-f938786ee383\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") "
Sep 16 13:20:00.481720 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.481611    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-tmp-dir\") pod \"031b78a8-d805-410d-a0b0-f938786ee383\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") "
Sep 16 13:20:00.481720 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.481669    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b78a8-d805-410d-a0b0-f938786ee383-tls-certs\") pod \"031b78a8-d805-410d-a0b0-f938786ee383\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") "
Sep 16 13:20:00.481720 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.481714    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc627\" (UniqueName: \"kubernetes.io/projected/031b78a8-d805-410d-a0b0-f938786ee383-kube-api-access-sc627\") pod \"031b78a8-d805-410d-a0b0-f938786ee383\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") "
Sep 16 13:20:00.481937 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.481754    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-dshm\") pod \"031b78a8-d805-410d-a0b0-f938786ee383\" (UID: \"031b78a8-d805-410d-a0b0-f938786ee383\") "
Sep 16 13:20:00.481937 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.481764    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-model-cache" (OuterVolumeSpecName: "model-cache") pod "031b78a8-d805-410d-a0b0-f938786ee383" (UID: "031b78a8-d805-410d-a0b0-f938786ee383"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:20:00.481937 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.481860    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-home" (OuterVolumeSpecName: "home") pod "031b78a8-d805-410d-a0b0-f938786ee383" (UID: "031b78a8-d805-410d-a0b0-f938786ee383"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:20:00.481937 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.481889    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "031b78a8-d805-410d-a0b0-f938786ee383" (UID: "031b78a8-d805-410d-a0b0-f938786ee383"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:20:00.482126 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.482012    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:20:00.482126 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.482030    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:20:00.482126 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.482043    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:20:00.484011 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.483984    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031b78a8-d805-410d-a0b0-f938786ee383-kube-api-access-sc627" (OuterVolumeSpecName: "kube-api-access-sc627") pod "031b78a8-d805-410d-a0b0-f938786ee383" (UID: "031b78a8-d805-410d-a0b0-f938786ee383"). InnerVolumeSpecName "kube-api-access-sc627". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:20:00.484011 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.483994    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-dshm" (OuterVolumeSpecName: "dshm") pod "031b78a8-d805-410d-a0b0-f938786ee383" (UID: "031b78a8-d805-410d-a0b0-f938786ee383"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:20:00.484144 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.484079    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031b78a8-d805-410d-a0b0-f938786ee383-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "031b78a8-d805-410d-a0b0-f938786ee383" (UID: "031b78a8-d805-410d-a0b0-f938786ee383"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:20:00.583260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.583181    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b78a8-d805-410d-a0b0-f938786ee383-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:20:00.583260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.583208    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc627\" (UniqueName: \"kubernetes.io/projected/031b78a8-d805-410d-a0b0-f938786ee383-kube-api-access-sc627\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:20:00.583260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:00.583218    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b78a8-d805-410d-a0b0-f938786ee383-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:20:01.042742 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.042706    2575 generic.go:356] "Generic (PLEG): container finished" podID="031b78a8-d805-410d-a0b0-f938786ee383" containerID="a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd" exitCode=0
Sep 16 13:20:01.042936 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.042748    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"
Sep 16 13:20:01.042936 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.042795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46" event={"ID":"031b78a8-d805-410d-a0b0-f938786ee383","Type":"ContainerDied","Data":"a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd"}
Sep 16 13:20:01.042936 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.042849    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46" event={"ID":"031b78a8-d805-410d-a0b0-f938786ee383","Type":"ContainerDied","Data":"849328be5fdcd69143a28a444d7aa5fac3fb39aced502fc76a48b40dadb34bc5"}
Sep 16 13:20:01.042936 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.042880    2575 scope.go:117] "RemoveContainer" containerID="a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd"
Sep 16 13:20:01.052398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.052380    2575 scope.go:117] "RemoveContainer" containerID="a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd"
Sep 16 13:20:01.052690 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:20:01.052669    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd\": container with ID starting with a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd not found: ID does not exist" containerID="a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd"
Sep 16 13:20:01.052757 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.052696    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd"} err="failed to get container status \"a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd\": rpc error: code = NotFound desc = could not find container \"a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd\": container with ID starting with a0d7a3c24374cf7fe95a523738641f3a370a20d4ce5faeb279ab0a5b313761cd not found: ID does not exist"
Sep 16 13:20:01.069593 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.069573    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"]
Sep 16 13:20:01.078999 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.078977    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-kmx46"]
Sep 16 13:20:01.334298 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.334219    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="031b78a8-d805-410d-a0b0-f938786ee383" path="/var/lib/kubelet/pods/031b78a8-d805-410d-a0b0-f938786ee383/volumes"
Sep 16 13:20:01.334589 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:01.334575    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0407135-c489-4689-869f-ce22d4e5e1e9" path="/var/lib/kubelet/pods/e0407135-c489-4689-869f-ce22d4e5e1e9/volumes"
Sep 16 13:20:02.629015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:02.628978    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:20:02.641887 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:02.641860    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:20:03.067155 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:03.067069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:20:07.369694 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:07.369635    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:07.524369 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:07.524337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:15.072092 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.072054    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"]
Sep 16 13:20:15.072436 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.072414    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031b78a8-d805-410d-a0b0-f938786ee383" containerName="main"
Sep 16 13:20:15.072436 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.072425    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="031b78a8-d805-410d-a0b0-f938786ee383" containerName="main"
Sep 16 13:20:15.072526 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.072447    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0407135-c489-4689-869f-ce22d4e5e1e9" containerName="main"
Sep 16 13:20:15.072526 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.072453    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0407135-c489-4689-869f-ce22d4e5e1e9" containerName="main"
Sep 16 13:20:15.072526 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.072506    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0407135-c489-4689-869f-ce22d4e5e1e9" containerName="main"
Sep 16 13:20:15.072526 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.072514    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="031b78a8-d805-410d-a0b0-f938786ee383" containerName="main"
Sep 16 13:20:15.075546 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.075530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.082492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.082467    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 13:20:15.098304 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.098281    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"]
Sep 16 13:20:15.203490 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.203465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-home\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.203663 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.203496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.203663 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.203516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41322ba7-ef9c-4c6e-b621-377ab2d852df-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.203663 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.203567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8qdp\" (UniqueName: \"kubernetes.io/projected/41322ba7-ef9c-4c6e-b621-377ab2d852df-kube-api-access-p8qdp\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.203663 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.203638    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.203807 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.203691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.301372 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.301330    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:15.304280 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.304256    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.304387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.304297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-home\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.304387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.304317    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.304387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.304334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41322ba7-ef9c-4c6e-b621-377ab2d852df-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.304387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.304355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8qdp\" (UniqueName: \"kubernetes.io/projected/41322ba7-ef9c-4c6e-b621-377ab2d852df-kube-api-access-p8qdp\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.304580 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.304389    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.304749 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.304725    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-home\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.304749 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.304745    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.304864 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.304818    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.306846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.306815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.306984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.306950    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41322ba7-ef9c-4c6e-b621-377ab2d852df-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.330833 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.330781    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8qdp\" (UniqueName: \"kubernetes.io/projected/41322ba7-ef9c-4c6e-b621-377ab2d852df-kube-api-access-p8qdp\") pod \"test-div-v2-kserve-5f6fdb6b77-wvlx8\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.385190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.385168    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:15.671157 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:15.671132    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"]
Sep 16 13:20:15.673284 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:20:15.673243    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41322ba7_ef9c_4c6e_b621_377ab2d852df.slice/crio-09da1a9cab309ceec9f2a60ace75bfa961d6bc21b51f1ca8f60a7e3e3faf4404 WatchSource:0}: Error finding container 09da1a9cab309ceec9f2a60ace75bfa961d6bc21b51f1ca8f60a7e3e3faf4404: Status 404 returned error can't find the container with id 09da1a9cab309ceec9f2a60ace75bfa961d6bc21b51f1ca8f60a7e3e3faf4404
Sep 16 13:20:16.129102 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:16.129064    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8" event={"ID":"41322ba7-ef9c-4c6e-b621-377ab2d852df","Type":"ContainerStarted","Data":"dd28b8c4277564006e79c9b9c9bdf4bf83439d0c4f690ec6be009c8c9b781a11"}
Sep 16 13:20:16.129461 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:16.129116    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8" event={"ID":"41322ba7-ef9c-4c6e-b621-377ab2d852df","Type":"ContainerStarted","Data":"09da1a9cab309ceec9f2a60ace75bfa961d6bc21b51f1ca8f60a7e3e3faf4404"}
Sep 16 13:20:16.160809 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:16.160766    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8" podStartSLOduration=1.16075082 podStartE2EDuration="1.16075082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:20:16.156941012 +0000 UTC m=+1937.338083664" watchObservedRunningTime="2026-09-16 13:20:16.16075082 +0000 UTC m=+1937.341893473"
Sep 16 13:20:16.385627 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:16.385551    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:16.661191 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:16.661117    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:24.499161 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:24.499122    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:24.613705 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:24.613675    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:25.385557 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:25.385525    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:25.398382 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:25.398360    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:26.198622 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:26.198586    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:20:28.232791 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:28.232756    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:29.068692 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.068662    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"]
Sep 16 13:20:29.072426 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.072407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.075460 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.075437    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 13:20:29.075580 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.075439    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 13:20:29.077220 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.076863    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-ls7zr\""
Sep 16 13:20:29.077220 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.077026    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 13:20:29.087877 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.087814    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"]
Sep 16 13:20:29.143278 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.143251    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:29.211328 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.211305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.211467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.211342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.211467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.211369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.211467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.211394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.211467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.211417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff458a-16f1-4897-b6c3-5e2badff1243-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.211467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.211452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s726m\" (UniqueName: \"kubernetes.io/projected/aeff458a-16f1-4897-b6c3-5e2badff1243-kube-api-access-s726m\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.312777 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.312751    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.313162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.312791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.313162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.312814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.313162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.312829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.313162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.312858    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff458a-16f1-4897-b6c3-5e2badff1243-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.313162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.312903    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s726m\" (UniqueName: \"kubernetes.io/projected/aeff458a-16f1-4897-b6c3-5e2badff1243-kube-api-access-s726m\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.313162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.313118    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.313429 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.313189    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.313429 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.313217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.315337 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.315320    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.315436 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.315419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff458a-16f1-4897-b6c3-5e2badff1243-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.323236 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.323181    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s726m\" (UniqueName: \"kubernetes.io/projected/aeff458a-16f1-4897-b6c3-5e2badff1243-kube-api-access-s726m\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.388100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.388064    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:29.453558 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.453530    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:29.667216 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:29.667189    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"]
Sep 16 13:20:29.667804 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:20:29.667781    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeff458a_16f1_4897_b6c3_5e2badff1243.slice/crio-bc156c0d26075a7a7818b98f21fd7c458540bbec8f400225e9fb249ed9ae3030 WatchSource:0}: Error finding container bc156c0d26075a7a7818b98f21fd7c458540bbec8f400225e9fb249ed9ae3030: Status 404 returned error can't find the container with id bc156c0d26075a7a7818b98f21fd7c458540bbec8f400225e9fb249ed9ae3030
Sep 16 13:20:30.210616 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:30.210581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh" event={"ID":"aeff458a-16f1-4897-b6c3-5e2badff1243","Type":"ContainerStarted","Data":"3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976"}
Sep 16 13:20:30.210802 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:30.210693    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh" event={"ID":"aeff458a-16f1-4897-b6c3-5e2badff1243","Type":"ContainerStarted","Data":"bc156c0d26075a7a7818b98f21fd7c458540bbec8f400225e9fb249ed9ae3030"}
Sep 16 13:20:30.231823 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:30.231778    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh" podStartSLOduration=1.231762797 podStartE2EDuration="1.231762797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:20:30.229040607 +0000 UTC m=+1951.410183252" watchObservedRunningTime="2026-09-16 13:20:30.231762797 +0000 UTC m=+1951.412905516"
Sep 16 13:20:30.389095 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:30.389069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:34.982595 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:34.982561    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:35.086835 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:35.086807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:20:39.388704 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:39.388676    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:39.401670 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:39.401634    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:40.273693 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:40.273659    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:20:54.008788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:20:54.008750    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:00.648540 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:00.648502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:00.795309 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:00.795273    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:00.924890 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:00.924803    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:01.628080 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:01.628041    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:01.630524 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:01.630498    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"]
Sep 16 13:21:01.829985 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:01.829945    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:02.274733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:02.274701    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"]
Sep 16 13:21:02.347694 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:02.347635    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:05.809414 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:05.809379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:06.454435 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.454403    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"]
Sep 16 13:21:06.457929 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.457911    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.462770 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.462746    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 13:21:06.464097 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.464065    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 13:21:06.464206 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.464096    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 16 13:21:06.464206 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.464124    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-zbgbw\""
Sep 16 13:21:06.497321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.497293    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"]
Sep 16 13:21:06.509398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.509374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:06.509503 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.509445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dgz\" (UniqueName: \"kubernetes.io/projected/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-kube-api-access-h2dgz\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.509503 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.509481    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tls-certs\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.509628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.509504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-dshm\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.509628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.509603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-home\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.509769 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.509686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.509769 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.509726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-model-cache\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.610124 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.610097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dgz\" (UniqueName: \"kubernetes.io/projected/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-kube-api-access-h2dgz\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.610296 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.610137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tls-certs\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.610296 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.610167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-dshm\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.610477 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.610340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-home\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.610477 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.610397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.610477 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.610424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-model-cache\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.610712 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.610695    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-home\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.610805 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.610743    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.611093 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.611048    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-model-cache\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.612511 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.612489    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-dshm\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.612913 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.612893    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tls-certs\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.635593 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.635570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dgz\" (UniqueName: \"kubernetes.io/projected/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-kube-api-access-h2dgz\") pod \"stop-v1-kserve-5657b88c95-sj5lf\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:06.768829 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:06.768764    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:07.046753 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:07.046728    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"]
Sep 16 13:21:07.047847 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:21:07.047816    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9ce365_d6c2_4e20_9f73_dd0d1c7078e3.slice/crio-94c1bda96cf79f9f8dbc895a79e68cd2e2ad8c5f68c96269255238ae10dcbbf0 WatchSource:0}: Error finding container 94c1bda96cf79f9f8dbc895a79e68cd2e2ad8c5f68c96269255238ae10dcbbf0: Status 404 returned error can't find the container with id 94c1bda96cf79f9f8dbc895a79e68cd2e2ad8c5f68c96269255238ae10dcbbf0
Sep 16 13:21:07.412046 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:07.412011    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf" event={"ID":"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3","Type":"ContainerStarted","Data":"68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829"}
Sep 16 13:21:07.412218 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:07.412071    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf" event={"ID":"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3","Type":"ContainerStarted","Data":"94c1bda96cf79f9f8dbc895a79e68cd2e2ad8c5f68c96269255238ae10dcbbf0"}
Sep 16 13:21:07.435782 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:07.435739    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf" podStartSLOduration=1.435720086 podStartE2EDuration="1.435720086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:07.433756235 +0000 UTC m=+1988.614898893" watchObservedRunningTime="2026-09-16 13:21:07.435720086 +0000 UTC m=+1988.616862739"
Sep 16 13:21:07.769779 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:07.769699    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:08.977003 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:08.976970    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:15.024331 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:15.024303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:15.094257 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:15.094223    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:16.649811 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.649755    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f" podUID="6030c021-13b8-44ab-a0f7-a32af63bb308" containerName="main" containerID="cri-o://9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6" gracePeriod=45
Sep 16 13:21:16.769759 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.769731    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:16.782881 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.782859    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:16.895889 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.895859    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:21:16.991045 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.990968    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-home\") pod \"6030c021-13b8-44ab-a0f7-a32af63bb308\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") "
Sep 16 13:21:16.991045 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991008    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-tmp-dir\") pod \"6030c021-13b8-44ab-a0f7-a32af63bb308\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") "
Sep 16 13:21:16.991045 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991042    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hxd2\" (UniqueName: \"kubernetes.io/projected/6030c021-13b8-44ab-a0f7-a32af63bb308-kube-api-access-7hxd2\") pod \"6030c021-13b8-44ab-a0f7-a32af63bb308\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") "
Sep 16 13:21:16.991322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991066    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-dshm\") pod \"6030c021-13b8-44ab-a0f7-a32af63bb308\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") "
Sep 16 13:21:16.991322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991085    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-model-cache\") pod \"6030c021-13b8-44ab-a0f7-a32af63bb308\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") "
Sep 16 13:21:16.991322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991111    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6030c021-13b8-44ab-a0f7-a32af63bb308-tls-certs\") pod \"6030c021-13b8-44ab-a0f7-a32af63bb308\" (UID: \"6030c021-13b8-44ab-a0f7-a32af63bb308\") "
Sep 16 13:21:16.991322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991254    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-home" (OuterVolumeSpecName: "home") pod "6030c021-13b8-44ab-a0f7-a32af63bb308" (UID: "6030c021-13b8-44ab-a0f7-a32af63bb308"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:16.991507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991375    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-model-cache" (OuterVolumeSpecName: "model-cache") pod "6030c021-13b8-44ab-a0f7-a32af63bb308" (UID: "6030c021-13b8-44ab-a0f7-a32af63bb308"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:16.991507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991394    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6030c021-13b8-44ab-a0f7-a32af63bb308" (UID: "6030c021-13b8-44ab-a0f7-a32af63bb308"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:16.991507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991461    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:16.991507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.991479    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:16.993448 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.993420    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-dshm" (OuterVolumeSpecName: "dshm") pod "6030c021-13b8-44ab-a0f7-a32af63bb308" (UID: "6030c021-13b8-44ab-a0f7-a32af63bb308"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:16.993448 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.993417    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6030c021-13b8-44ab-a0f7-a32af63bb308-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6030c021-13b8-44ab-a0f7-a32af63bb308" (UID: "6030c021-13b8-44ab-a0f7-a32af63bb308"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:21:16.993599 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:16.993452    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6030c021-13b8-44ab-a0f7-a32af63bb308-kube-api-access-7hxd2" (OuterVolumeSpecName: "kube-api-access-7hxd2") pod "6030c021-13b8-44ab-a0f7-a32af63bb308" (UID: "6030c021-13b8-44ab-a0f7-a32af63bb308"). InnerVolumeSpecName "kube-api-access-7hxd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:21:17.092653 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.092617    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hxd2\" (UniqueName: \"kubernetes.io/projected/6030c021-13b8-44ab-a0f7-a32af63bb308-kube-api-access-7hxd2\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.092771 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.092668    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.092771 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.092679    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6030c021-13b8-44ab-a0f7-a32af63bb308-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.092771 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.092688    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6030c021-13b8-44ab-a0f7-a32af63bb308-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.294292 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.294218    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8" podUID="41322ba7-ef9c-4c6e-b621-377ab2d852df" containerName="main" containerID="cri-o://dd28b8c4277564006e79c9b9c9bdf4bf83439d0c4f690ec6be009c8c9b781a11" gracePeriod=45
Sep 16 13:21:17.383237 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.383205    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"]
Sep 16 13:21:17.383626 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.383608    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6030c021-13b8-44ab-a0f7-a32af63bb308" containerName="main"
Sep 16 13:21:17.383626 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.383624    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6030c021-13b8-44ab-a0f7-a32af63bb308" containerName="main"
Sep 16 13:21:17.383814 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.383720    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6030c021-13b8-44ab-a0f7-a32af63bb308" containerName="main"
Sep 16 13:21:17.386761 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.386740    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.390180 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.390161    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 16 13:21:17.398556 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.398531    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"]
Sep 16 13:21:17.427590 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.427559    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:17.465763 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.465738    2575 generic.go:356] "Generic (PLEG): container finished" podID="41322ba7-ef9c-4c6e-b621-377ab2d852df" containerID="dd28b8c4277564006e79c9b9c9bdf4bf83439d0c4f690ec6be009c8c9b781a11" exitCode=0
Sep 16 13:21:17.465901 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.465813    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8" event={"ID":"41322ba7-ef9c-4c6e-b621-377ab2d852df","Type":"ContainerDied","Data":"dd28b8c4277564006e79c9b9c9bdf4bf83439d0c4f690ec6be009c8c9b781a11"}
Sep 16 13:21:17.467213 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.467192    2575 generic.go:356] "Generic (PLEG): container finished" podID="6030c021-13b8-44ab-a0f7-a32af63bb308" containerID="9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6" exitCode=0
Sep 16 13:21:17.467317 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.467235    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"
Sep 16 13:21:17.467317 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.467278    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f" event={"ID":"6030c021-13b8-44ab-a0f7-a32af63bb308","Type":"ContainerDied","Data":"9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6"}
Sep 16 13:21:17.467412 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.467330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f" event={"ID":"6030c021-13b8-44ab-a0f7-a32af63bb308","Type":"ContainerDied","Data":"daa7ddda177936fbbfb9b09490dea42a9879276eb463ac098a85d724a31cb530"}
Sep 16 13:21:17.467412 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.467358    2575 scope.go:117] "RemoveContainer" containerID="9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6"
Sep 16 13:21:17.477958 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.477937    2575 scope.go:117] "RemoveContainer" containerID="9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6"
Sep 16 13:21:17.478353 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:21:17.478328    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6\": container with ID starting with 9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6 not found: ID does not exist" containerID="9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6"
Sep 16 13:21:17.478453 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.478366    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6"} err="failed to get container status \"9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6\": rpc error: code = NotFound desc = could not find container \"9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6\": container with ID starting with 9aa57b864d16954ade577e6dc46ee0297d34a38e90c487c2bd9388bbf1ca91c6 not found: ID does not exist"
Sep 16 13:21:17.480995 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.480974    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:21:17.489892 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.489871    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"]
Sep 16 13:21:17.496592 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.496571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9lhn\" (UniqueName: \"kubernetes.io/projected/3cc8515b-a98e-42e7-922c-6790f1ec2963-kube-api-access-t9lhn\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.496935 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.496588    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-kb45f"]
Sep 16 13:21:17.496935 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.496603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-model-cache\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.496935 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.496695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-dshm\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.496935 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.496736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.496935 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.496801    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-home\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.496935 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.496859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc8515b-a98e-42e7-922c-6790f1ec2963-tls-certs\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.544986 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.544930    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:21:17.597874 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.597838    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-tmp-dir\") pod \"41322ba7-ef9c-4c6e-b621-377ab2d852df\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") "
Sep 16 13:21:17.598053 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.597895    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8qdp\" (UniqueName: \"kubernetes.io/projected/41322ba7-ef9c-4c6e-b621-377ab2d852df-kube-api-access-p8qdp\") pod \"41322ba7-ef9c-4c6e-b621-377ab2d852df\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") "
Sep 16 13:21:17.598053 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.597930    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-home\") pod \"41322ba7-ef9c-4c6e-b621-377ab2d852df\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") "
Sep 16 13:21:17.598053 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.597955    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-model-cache\") pod \"41322ba7-ef9c-4c6e-b621-377ab2d852df\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") "
Sep 16 13:21:17.598053 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.597974    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-dshm\") pod \"41322ba7-ef9c-4c6e-b621-377ab2d852df\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") "
Sep 16 13:21:17.598053 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598018    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41322ba7-ef9c-4c6e-b621-377ab2d852df-tls-certs\") pod \"41322ba7-ef9c-4c6e-b621-377ab2d852df\" (UID: \"41322ba7-ef9c-4c6e-b621-377ab2d852df\") "
Sep 16 13:21:17.598322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598149    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41322ba7-ef9c-4c6e-b621-377ab2d852df" (UID: "41322ba7-ef9c-4c6e-b621-377ab2d852df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:17.598322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9lhn\" (UniqueName: \"kubernetes.io/projected/3cc8515b-a98e-42e7-922c-6790f1ec2963-kube-api-access-t9lhn\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.598322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-model-cache\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.598322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598219    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-home" (OuterVolumeSpecName: "home") pod "41322ba7-ef9c-4c6e-b621-377ab2d852df" (UID: "41322ba7-ef9c-4c6e-b621-377ab2d852df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:17.598322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598233    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-dshm\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.598322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.598628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-home\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.598628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc8515b-a98e-42e7-922c-6790f1ec2963-tls-certs\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.598628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598442    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-model-cache" (OuterVolumeSpecName: "model-cache") pod "41322ba7-ef9c-4c6e-b621-377ab2d852df" (UID: "41322ba7-ef9c-4c6e-b621-377ab2d852df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:17.598628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598462    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.598628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598476    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.598628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598558    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-model-cache\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.598989 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.598989 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.598854    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-home\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.600502 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.600480    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41322ba7-ef9c-4c6e-b621-377ab2d852df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41322ba7-ef9c-4c6e-b621-377ab2d852df" (UID: "41322ba7-ef9c-4c6e-b621-377ab2d852df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:21:17.600611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.600582    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-dshm" (OuterVolumeSpecName: "dshm") pod "41322ba7-ef9c-4c6e-b621-377ab2d852df" (UID: "41322ba7-ef9c-4c6e-b621-377ab2d852df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:17.600808 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.600782    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41322ba7-ef9c-4c6e-b621-377ab2d852df-kube-api-access-p8qdp" (OuterVolumeSpecName: "kube-api-access-p8qdp") pod "41322ba7-ef9c-4c6e-b621-377ab2d852df" (UID: "41322ba7-ef9c-4c6e-b621-377ab2d852df"). InnerVolumeSpecName "kube-api-access-p8qdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:21:17.600926 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.600873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc8515b-a98e-42e7-922c-6790f1ec2963-tls-certs\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.600926 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.600912    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-dshm\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.607314 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.607297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9lhn\" (UniqueName: \"kubernetes.io/projected/3cc8515b-a98e-42e7-922c-6790f1ec2963-kube-api-access-t9lhn\") pod \"stop-v2-kserve-7f68747c45-nxr9c\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.697954 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.697926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:17.699198 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.699178    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8qdp\" (UniqueName: \"kubernetes.io/projected/41322ba7-ef9c-4c6e-b621-377ab2d852df-kube-api-access-p8qdp\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.699252 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.699200    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.699252 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.699209    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41322ba7-ef9c-4c6e-b621-377ab2d852df-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.699252 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.699217    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41322ba7-ef9c-4c6e-b621-377ab2d852df-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:17.997437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:17.997412    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"]
Sep 16 13:21:17.999451 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:21:17.999422    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc8515b_a98e_42e7_922c_6790f1ec2963.slice/crio-edcb8fdd453d96cacc1614fb01789af7f9b553e9226ff9bc6a40e15e219384a9 WatchSource:0}: Error finding container edcb8fdd453d96cacc1614fb01789af7f9b553e9226ff9bc6a40e15e219384a9: Status 404 returned error can't find the container with id edcb8fdd453d96cacc1614fb01789af7f9b553e9226ff9bc6a40e15e219384a9
Sep 16 13:21:18.475100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:18.475053    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c" event={"ID":"3cc8515b-a98e-42e7-922c-6790f1ec2963","Type":"ContainerStarted","Data":"675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e"}
Sep 16 13:21:18.475292 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:18.475113    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c" event={"ID":"3cc8515b-a98e-42e7-922c-6790f1ec2963","Type":"ContainerStarted","Data":"edcb8fdd453d96cacc1614fb01789af7f9b553e9226ff9bc6a40e15e219384a9"}
Sep 16 13:21:18.477549 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:18.477517    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8" event={"ID":"41322ba7-ef9c-4c6e-b621-377ab2d852df","Type":"ContainerDied","Data":"09da1a9cab309ceec9f2a60ace75bfa961d6bc21b51f1ca8f60a7e3e3faf4404"}
Sep 16 13:21:18.477549 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:18.477539    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"
Sep 16 13:21:18.477760 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:18.477582    2575 scope.go:117] "RemoveContainer" containerID="dd28b8c4277564006e79c9b9c9bdf4bf83439d0c4f690ec6be009c8c9b781a11"
Sep 16 13:21:18.498303 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:18.498257    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c" podStartSLOduration=1.4982430500000001 podStartE2EDuration="1.49824305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:21:18.494882219 +0000 UTC m=+1999.676024871" watchObservedRunningTime="2026-09-16 13:21:18.49824305 +0000 UTC m=+1999.679385703"
Sep 16 13:21:18.520121 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:18.520094    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"]
Sep 16 13:21:18.522093 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:18.522077    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-wvlx8"]
Sep 16 13:21:18.698738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:18.698705    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:19.329850 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:19.329818    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41322ba7-ef9c-4c6e-b621-377ab2d852df" path="/var/lib/kubelet/pods/41322ba7-ef9c-4c6e-b621-377ab2d852df/volumes"
Sep 16 13:21:19.330201 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:19.330186    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6030c021-13b8-44ab-a0f7-a32af63bb308" path="/var/lib/kubelet/pods/6030c021-13b8-44ab-a0f7-a32af63bb308/volumes"
Sep 16 13:21:19.450102 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:19.450066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:19.471353 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:19.471321    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"]
Sep 16 13:21:19.480390 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:21:19.480359    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:21:19.480390 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:21:19.480390 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:21:19.480390 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976" cmd=["/bin/sleep","15"]
Sep 16 13:21:19.480609 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:19.480401    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh" message="" err=<
Sep 16 13:21:19.480609 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:21:19.480609 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:21:19.480609 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:21:19.480609 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:21:19.480413    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:21:19.480609 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:21:19.480609 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:21:19.480609 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh" podUID="aeff458a-16f1-4897-b6c3-5e2badff1243" containerName="main" containerID="cri-o://3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976"
Sep 16 13:21:19.480609 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:19.480449    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh" podUID="aeff458a-16f1-4897-b6c3-5e2badff1243" containerName="main" containerID="cri-o://3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976" gracePeriod=60
Sep 16 13:21:19.544240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:19.544214    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:20.236967 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.236942    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:21:20.321693 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.321665    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff458a-16f1-4897-b6c3-5e2badff1243-tls-certs\") pod \"aeff458a-16f1-4897-b6c3-5e2badff1243\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") "
Sep 16 13:21:20.321871 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.321718    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s726m\" (UniqueName: \"kubernetes.io/projected/aeff458a-16f1-4897-b6c3-5e2badff1243-kube-api-access-s726m\") pod \"aeff458a-16f1-4897-b6c3-5e2badff1243\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") "
Sep 16 13:21:20.321871 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.321771    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-dshm\") pod \"aeff458a-16f1-4897-b6c3-5e2badff1243\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") "
Sep 16 13:21:20.321871 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.321810    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-home\") pod \"aeff458a-16f1-4897-b6c3-5e2badff1243\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") "
Sep 16 13:21:20.322046 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.321879    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-model-cache\") pod \"aeff458a-16f1-4897-b6c3-5e2badff1243\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") "
Sep 16 13:21:20.322046 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.321936    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-tmp-dir\") pod \"aeff458a-16f1-4897-b6c3-5e2badff1243\" (UID: \"aeff458a-16f1-4897-b6c3-5e2badff1243\") "
Sep 16 13:21:20.322140 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.322076    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-home" (OuterVolumeSpecName: "home") pod "aeff458a-16f1-4897-b6c3-5e2badff1243" (UID: "aeff458a-16f1-4897-b6c3-5e2badff1243"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:20.322196 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.322155    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-model-cache" (OuterVolumeSpecName: "model-cache") pod "aeff458a-16f1-4897-b6c3-5e2badff1243" (UID: "aeff458a-16f1-4897-b6c3-5e2badff1243"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:20.322251 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.322217    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.322347 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.322285    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aeff458a-16f1-4897-b6c3-5e2badff1243" (UID: "aeff458a-16f1-4897-b6c3-5e2badff1243"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:20.324154 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.324129    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeff458a-16f1-4897-b6c3-5e2badff1243-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aeff458a-16f1-4897-b6c3-5e2badff1243" (UID: "aeff458a-16f1-4897-b6c3-5e2badff1243"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:21:20.324414 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.324391    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-dshm" (OuterVolumeSpecName: "dshm") pod "aeff458a-16f1-4897-b6c3-5e2badff1243" (UID: "aeff458a-16f1-4897-b6c3-5e2badff1243"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:21:20.324414 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.324404    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeff458a-16f1-4897-b6c3-5e2badff1243-kube-api-access-s726m" (OuterVolumeSpecName: "kube-api-access-s726m") pod "aeff458a-16f1-4897-b6c3-5e2badff1243" (UID: "aeff458a-16f1-4897-b6c3-5e2badff1243"). InnerVolumeSpecName "kube-api-access-s726m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:21:20.422963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.422935    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.422963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.422958    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.422963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.422967    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeff458a-16f1-4897-b6c3-5e2badff1243-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.423129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.422977    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s726m\" (UniqueName: \"kubernetes.io/projected/aeff458a-16f1-4897-b6c3-5e2badff1243-kube-api-access-s726m\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.423129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.422986    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeff458a-16f1-4897-b6c3-5e2badff1243-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:21:20.494154 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.494125    2575 generic.go:356] "Generic (PLEG): container finished" podID="aeff458a-16f1-4897-b6c3-5e2badff1243" containerID="3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976" exitCode=0
Sep 16 13:21:20.494255 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.494176    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"
Sep 16 13:21:20.494255 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.494208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh" event={"ID":"aeff458a-16f1-4897-b6c3-5e2badff1243","Type":"ContainerDied","Data":"3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976"}
Sep 16 13:21:20.494355 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.494276    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh" event={"ID":"aeff458a-16f1-4897-b6c3-5e2badff1243","Type":"ContainerDied","Data":"bc156c0d26075a7a7818b98f21fd7c458540bbec8f400225e9fb249ed9ae3030"}
Sep 16 13:21:20.494355 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.494312    2575 scope.go:117] "RemoveContainer" containerID="3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976"
Sep 16 13:21:20.504444 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.504426    2575 scope.go:117] "RemoveContainer" containerID="3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976"
Sep 16 13:21:20.504742 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:21:20.504725    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976\": container with ID starting with 3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976 not found: ID does not exist" containerID="3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976"
Sep 16 13:21:20.504809 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.504759    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976"} err="failed to get container status \"3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976\": rpc error: code = NotFound desc = could not find container \"3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976\": container with ID starting with 3931ddf2971a735426d3bfcbeb241315d48cfac264a9d93c2c9e1601e2419976 not found: ID does not exist"
Sep 16 13:21:20.519854 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.519832    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"]
Sep 16 13:21:20.528872 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:20.528851    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444mmjhh"]
Sep 16 13:21:21.329826 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:21.329791    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeff458a-16f1-4897-b6c3-5e2badff1243" path="/var/lib/kubelet/pods/aeff458a-16f1-4897-b6c3-5e2badff1243/volumes"
Sep 16 13:21:27.698846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:27.698805    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:27.711654 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:27.711614    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:28.550524 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:28.550492    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:21:43.376018 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:43.375984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:49.299621 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:49.299584    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:49.367907 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:49.367874    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:21:53.863034 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:53.863003    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"]
Sep 16 13:21:53.893618 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:21:53.893585    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:06.270772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:06.270729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:06.364660 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:06.364606    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:08.888572 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:08.888532    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c" podUID="3cc8515b-a98e-42e7-922c-6790f1ec2963" containerName="main" containerID="cri-o://675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e" gracePeriod=45
Sep 16 13:22:09.147421 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.147368    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:22:09.213318 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213295    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-home\") pod \"3cc8515b-a98e-42e7-922c-6790f1ec2963\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") "
Sep 16 13:22:09.213426 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213345    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-dshm\") pod \"3cc8515b-a98e-42e7-922c-6790f1ec2963\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") "
Sep 16 13:22:09.213426 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213374    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-model-cache\") pod \"3cc8515b-a98e-42e7-922c-6790f1ec2963\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") "
Sep 16 13:22:09.213426 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213395    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc8515b-a98e-42e7-922c-6790f1ec2963-tls-certs\") pod \"3cc8515b-a98e-42e7-922c-6790f1ec2963\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") "
Sep 16 13:22:09.213426 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213417    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9lhn\" (UniqueName: \"kubernetes.io/projected/3cc8515b-a98e-42e7-922c-6790f1ec2963-kube-api-access-t9lhn\") pod \"3cc8515b-a98e-42e7-922c-6790f1ec2963\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") "
Sep 16 13:22:09.213667 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213439    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-tmp-dir\") pod \"3cc8515b-a98e-42e7-922c-6790f1ec2963\" (UID: \"3cc8515b-a98e-42e7-922c-6790f1ec2963\") "
Sep 16 13:22:09.213667 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213563    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-home" (OuterVolumeSpecName: "home") pod "3cc8515b-a98e-42e7-922c-6790f1ec2963" (UID: "3cc8515b-a98e-42e7-922c-6790f1ec2963"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:09.213788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213681    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:09.213788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213698    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-model-cache" (OuterVolumeSpecName: "model-cache") pod "3cc8515b-a98e-42e7-922c-6790f1ec2963" (UID: "3cc8515b-a98e-42e7-922c-6790f1ec2963"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:09.213887 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.213790    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3cc8515b-a98e-42e7-922c-6790f1ec2963" (UID: "3cc8515b-a98e-42e7-922c-6790f1ec2963"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:09.215748 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.215729    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-dshm" (OuterVolumeSpecName: "dshm") pod "3cc8515b-a98e-42e7-922c-6790f1ec2963" (UID: "3cc8515b-a98e-42e7-922c-6790f1ec2963"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:09.215748 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.215736    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc8515b-a98e-42e7-922c-6790f1ec2963-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3cc8515b-a98e-42e7-922c-6790f1ec2963" (UID: "3cc8515b-a98e-42e7-922c-6790f1ec2963"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:22:09.215856 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.215794    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc8515b-a98e-42e7-922c-6790f1ec2963-kube-api-access-t9lhn" (OuterVolumeSpecName: "kube-api-access-t9lhn") pod "3cc8515b-a98e-42e7-922c-6790f1ec2963" (UID: "3cc8515b-a98e-42e7-922c-6790f1ec2963"). InnerVolumeSpecName "kube-api-access-t9lhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:22:09.314620 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.314599    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9lhn\" (UniqueName: \"kubernetes.io/projected/3cc8515b-a98e-42e7-922c-6790f1ec2963-kube-api-access-t9lhn\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:09.314620 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.314618    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:09.314758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.314628    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:09.314758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.314636    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc8515b-a98e-42e7-922c-6790f1ec2963-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:09.314758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.314664    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc8515b-a98e-42e7-922c-6790f1ec2963-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:09.759256 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.759227    2575 generic.go:356] "Generic (PLEG): container finished" podID="3cc8515b-a98e-42e7-922c-6790f1ec2963" containerID="675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e" exitCode=0
Sep 16 13:22:09.759425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.759264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c" event={"ID":"3cc8515b-a98e-42e7-922c-6790f1ec2963","Type":"ContainerDied","Data":"675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e"}
Sep 16 13:22:09.759425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.759280    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"
Sep 16 13:22:09.759425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.759299    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c" event={"ID":"3cc8515b-a98e-42e7-922c-6790f1ec2963","Type":"ContainerDied","Data":"edcb8fdd453d96cacc1614fb01789af7f9b553e9226ff9bc6a40e15e219384a9"}
Sep 16 13:22:09.759425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.759326    2575 scope.go:117] "RemoveContainer" containerID="675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e"
Sep 16 13:22:09.768557 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.768544    2575 scope.go:117] "RemoveContainer" containerID="675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e"
Sep 16 13:22:09.768839 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:22:09.768819    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e\": container with ID starting with 675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e not found: ID does not exist" containerID="675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e"
Sep 16 13:22:09.768888 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.768849    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e"} err="failed to get container status \"675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e\": rpc error: code = NotFound desc = could not find container \"675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e\": container with ID starting with 675c33aa4186dc87299ad1dfcff03ca1e85906488c2385ae79b8feb651dd2a2e not found: ID does not exist"
Sep 16 13:22:09.776236 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.776214    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"]
Sep 16 13:22:09.780195 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:09.780175    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-nxr9c"]
Sep 16 13:22:11.328930 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:11.328899    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cc8515b-a98e-42e7-922c-6790f1ec2963" path="/var/lib/kubelet/pods/3cc8515b-a98e-42e7-922c-6790f1ec2963/volumes"
Sep 16 13:22:13.142333 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:13.142303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:15.545901 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:15.545862    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:15.644535 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:15.644502    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:15.739979 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:15.739944    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:19.040545 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:19.040516    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:19.141560 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:19.141531    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:20.522322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:20.522252    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"]
Sep 16 13:22:20.616668 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:20.616613    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:21.919380 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919351    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"]
Sep 16 13:22:21.919767 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919710    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41322ba7-ef9c-4c6e-b621-377ab2d852df" containerName="main"
Sep 16 13:22:21.919767 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919721    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="41322ba7-ef9c-4c6e-b621-377ab2d852df" containerName="main"
Sep 16 13:22:21.919767 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919734    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeff458a-16f1-4897-b6c3-5e2badff1243" containerName="main"
Sep 16 13:22:21.919767 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919740    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeff458a-16f1-4897-b6c3-5e2badff1243" containerName="main"
Sep 16 13:22:21.919767 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919759    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc8515b-a98e-42e7-922c-6790f1ec2963" containerName="main"
Sep 16 13:22:21.919767 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919766    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc8515b-a98e-42e7-922c-6790f1ec2963" containerName="main"
Sep 16 13:22:21.919962 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919822    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="41322ba7-ef9c-4c6e-b621-377ab2d852df" containerName="main"
Sep 16 13:22:21.919962 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919829    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeff458a-16f1-4897-b6c3-5e2badff1243" containerName="main"
Sep 16 13:22:21.919962 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.919838    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc8515b-a98e-42e7-922c-6790f1ec2963" containerName="main"
Sep 16 13:22:21.925419 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.925390    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:21.928785 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.928752    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-5rj8s\""
Sep 16 13:22:21.928785 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.928752    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 13:22:21.929999 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.929977    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:22:21.929999 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.929993    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 13:22:21.930188 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.930002    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 13:22:21.930188 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.930012    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 13:22:21.934688 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.934668    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"]
Sep 16 13:22:21.949547 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:21.949520    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:22.010557 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.010536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpncw\" (UniqueName: \"kubernetes.io/projected/5952519d-4a0c-4886-9009-1608cc96d915-kube-api-access-mpncw\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.010711 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.010573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-model-cache\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.010711 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.010594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-dshm\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.010830 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.010739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5952519d-4a0c-4886-9009-1608cc96d915-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.010830 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.010782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.010830 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.010825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.010956 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.010852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5952519d-4a0c-4886-9009-1608cc96d915-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.010956 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.010878    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-home\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.111814 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.111791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-model-cache\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.111918 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.111828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-dshm\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.111918 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.111874    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5952519d-4a0c-4886-9009-1608cc96d915-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.111918 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.111906    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.112104 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.111937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.112104 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.111975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5952519d-4a0c-4886-9009-1608cc96d915-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.112104 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.112001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-home\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.112104 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.112043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpncw\" (UniqueName: \"kubernetes.io/projected/5952519d-4a0c-4886-9009-1608cc96d915-kube-api-access-mpncw\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.112303 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.112195    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-model-cache\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.112303 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.112247    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.112303 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.112292    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.112546 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.112524    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-home\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.112842 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.112575    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5952519d-4a0c-4886-9009-1608cc96d915-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.114321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.114296    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-dshm\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.114554 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.114534    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5952519d-4a0c-4886-9009-1608cc96d915-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.121067 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.121048    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpncw\" (UniqueName: \"kubernetes.io/projected/5952519d-4a0c-4886-9009-1608cc96d915-kube-api-access-mpncw\") pod \"tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.213554 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.213485    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"]
Sep 16 13:22:22.218573 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.218555    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:22.221324 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.221263    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-s9lbf\""
Sep 16 13:22:22.225201 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.225178    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"]
Sep 16 13:22:22.227928 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.227868    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:22.237394 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.237366    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:22.314104 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.314076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2264f664-ddd7-4124-9743-557d6b99c8f3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf\" (UID: \"2264f664-ddd7-4124-9743-557d6b99c8f3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:22.314256 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.314178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wpw6\" (UniqueName: \"kubernetes.io/projected/2264f664-ddd7-4124-9743-557d6b99c8f3-kube-api-access-6wpw6\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf\" (UID: \"2264f664-ddd7-4124-9743-557d6b99c8f3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:22.327100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.327071    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:22.415559 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.415529    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2264f664-ddd7-4124-9743-557d6b99c8f3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf\" (UID: \"2264f664-ddd7-4124-9743-557d6b99c8f3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:22.415741 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.415610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wpw6\" (UniqueName: \"kubernetes.io/projected/2264f664-ddd7-4124-9743-557d6b99c8f3-kube-api-access-6wpw6\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf\" (UID: \"2264f664-ddd7-4124-9743-557d6b99c8f3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:22.418349 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.418328    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2264f664-ddd7-4124-9743-557d6b99c8f3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf\" (UID: \"2264f664-ddd7-4124-9743-557d6b99c8f3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:22.425900 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.425876    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wpw6\" (UniqueName: \"kubernetes.io/projected/2264f664-ddd7-4124-9743-557d6b99c8f3-kube-api-access-6wpw6\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf\" (UID: \"2264f664-ddd7-4124-9743-557d6b99c8f3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:22.518131 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.518110    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"]
Sep 16 13:22:22.519400 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:22:22.519378    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5952519d_4a0c_4886_9009_1608cc96d915.slice/crio-18e9d95dda77aa40cd0373165d4bfe2064217d1698e8d6d2cd77873e3401ce81 WatchSource:0}: Error finding container 18e9d95dda77aa40cd0373165d4bfe2064217d1698e8d6d2cd77873e3401ce81: Status 404 returned error can't find the container with id 18e9d95dda77aa40cd0373165d4bfe2064217d1698e8d6d2cd77873e3401ce81
Sep 16 13:22:22.531684 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.531664    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:22.808236 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.808208    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"]
Sep 16 13:22:22.808801 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:22:22.808774    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2264f664_ddd7_4124_9743_557d6b99c8f3.slice/crio-83a325a5dafa89021d20553994ca1b61d53a3f240998018a339b0bd6fc32c83f WatchSource:0}: Error finding container 83a325a5dafa89021d20553994ca1b61d53a3f240998018a339b0bd6fc32c83f: Status 404 returned error can't find the container with id 83a325a5dafa89021d20553994ca1b61d53a3f240998018a339b0bd6fc32c83f
Sep 16 13:22:22.829153 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.829125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" event={"ID":"2264f664-ddd7-4124-9743-557d6b99c8f3","Type":"ContainerStarted","Data":"83a325a5dafa89021d20553994ca1b61d53a3f240998018a339b0bd6fc32c83f"}
Sep 16 13:22:22.831062 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.831034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" event={"ID":"5952519d-4a0c-4886-9009-1608cc96d915","Type":"ContainerStarted","Data":"d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70"}
Sep 16 13:22:22.831187 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:22.831093    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" event={"ID":"5952519d-4a0c-4886-9009-1608cc96d915","Type":"ContainerStarted","Data":"18e9d95dda77aa40cd0373165d4bfe2064217d1698e8d6d2cd77873e3401ce81"}
Sep 16 13:22:23.839582 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:23.839547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" event={"ID":"2264f664-ddd7-4124-9743-557d6b99c8f3","Type":"ContainerStarted","Data":"fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9"}
Sep 16 13:22:23.840079 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:23.840007    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:25.853572 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:25.853539    2575 generic.go:356] "Generic (PLEG): container finished" podID="5952519d-4a0c-4886-9009-1608cc96d915" containerID="d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70" exitCode=0
Sep 16 13:22:25.854077 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:25.853633    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" event={"ID":"5952519d-4a0c-4886-9009-1608cc96d915","Type":"ContainerDied","Data":"d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70"}
Sep 16 13:22:25.883676 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:25.883601    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" podStartSLOduration=3.883579104 podStartE2EDuration="3.883579104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:23.869976463 +0000 UTC m=+2065.051119115" watchObservedRunningTime="2026-09-16 13:22:25.883579104 +0000 UTC m=+2067.064721761"
Sep 16 13:22:25.893377 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:25.893347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:26.862707 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:26.862666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" event={"ID":"5952519d-4a0c-4886-9009-1608cc96d915","Type":"ContainerStarted","Data":"d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72"}
Sep 16 13:22:26.892629 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:26.892579    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" podStartSLOduration=3.475707162 podStartE2EDuration="5.892564977s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.416857815s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:26.892068671 +0000 UTC m=+2068.073211326" watchObservedRunningTime="2026-09-16 13:22:26.892564977 +0000 UTC m=+2068.073707630"
Sep 16 13:22:27.237782 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:27.237703    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:32.237887 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:32.237853    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:32.250455 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:32.250432    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:32.904573 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:32.904548    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:22:35.542437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.542381    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf" podUID="8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" containerName="main" containerID="cri-o://68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829" gracePeriod=45
Sep 16 13:22:35.800419 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.800369    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:22:35.910105 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.910069    2575 generic.go:356] "Generic (PLEG): container finished" podID="8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" containerID="68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829" exitCode=0
Sep 16 13:22:35.910278 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.910118    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf" event={"ID":"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3","Type":"ContainerDied","Data":"68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829"}
Sep 16 13:22:35.910278 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.910120    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"
Sep 16 13:22:35.910278 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.910190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf" event={"ID":"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3","Type":"ContainerDied","Data":"94c1bda96cf79f9f8dbc895a79e68cd2e2ad8c5f68c96269255238ae10dcbbf0"}
Sep 16 13:22:35.910278 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.910214    2575 scope.go:117] "RemoveContainer" containerID="68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829"
Sep 16 13:22:35.918923 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.918906    2575 scope.go:117] "RemoveContainer" containerID="68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829"
Sep 16 13:22:35.919206 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:22:35.919186    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829\": container with ID starting with 68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829 not found: ID does not exist" containerID="68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829"
Sep 16 13:22:35.919246 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.919216    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829"} err="failed to get container status \"68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829\": rpc error: code = NotFound desc = could not find container \"68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829\": container with ID starting with 68fbe8bba9429b3e35a1e14e94a420a2e376dfad1eaf4a0a80e5bbf2232aa829 not found: ID does not exist"
Sep 16 13:22:35.923492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.923475    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-model-cache\") pod \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") "
Sep 16 13:22:35.923546 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.923512    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tls-certs\") pod \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") "
Sep 16 13:22:35.923546 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.923529    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tmp-dir\") pod \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") "
Sep 16 13:22:35.923698 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.923669    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2dgz\" (UniqueName: \"kubernetes.io/projected/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-kube-api-access-h2dgz\") pod \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") "
Sep 16 13:22:35.923780 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.923721    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-home\") pod \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") "
Sep 16 13:22:35.923780 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.923751    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" (UID: "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:35.923780 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.923766    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-dshm\") pod \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\" (UID: \"8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3\") "
Sep 16 13:22:35.923949 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.923779    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" (UID: "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:35.923949 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.923907    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-home" (OuterVolumeSpecName: "home") pod "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" (UID: "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:35.924045 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.924025    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:35.924082 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.924044    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:35.924082 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.924079    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:35.925813 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.925787    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" (UID: "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:22:35.925910 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.925870    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-kube-api-access-h2dgz" (OuterVolumeSpecName: "kube-api-access-h2dgz") pod "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" (UID: "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3"). InnerVolumeSpecName "kube-api-access-h2dgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:22:35.925958 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:35.925915    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-dshm" (OuterVolumeSpecName: "dshm") pod "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" (UID: "8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:22:36.025157 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:36.025123    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:36.025157 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:36.025155    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dgz\" (UniqueName: \"kubernetes.io/projected/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-kube-api-access-h2dgz\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:36.025157 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:36.025164    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:22:36.233246 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:36.233221    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"]
Sep 16 13:22:36.238273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:36.238252    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-sj5lf"]
Sep 16 13:22:37.329969 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:37.329931    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" path="/var/lib/kubelet/pods/8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3/volumes"
Sep 16 13:22:38.088110 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.088080    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"]
Sep 16 13:22:38.088474 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.088462    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" containerName="main"
Sep 16 13:22:38.088524 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.088475    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" containerName="main"
Sep 16 13:22:38.088564 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.088545    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9ce365-d6c2-4e20-9f73-dd0d1c7078e3" containerName="main"
Sep 16 13:22:38.091761 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.091742    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.096584 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.096560    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 13:22:38.096584 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.096575    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-mwhj2\""
Sep 16 13:22:38.096797 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.096594    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 13:22:38.096905 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.096884    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 13:22:38.102428 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.102368    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"]
Sep 16 13:22:38.120988 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.120952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:38.143992 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.143968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-model-cache\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.144098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.143999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-dshm\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.144098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.144018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4ssk\" (UniqueName: \"kubernetes.io/projected/41cc10dc-2174-4e66-a69c-be33e0fc9519-kube-api-access-j4ssk\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.144098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.144088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc10dc-2174-4e66-a69c-be33e0fc9519-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.144215 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.144122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-home\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.144215 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.144160    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.245539 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.245510    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-home\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.245624 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.245577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.245624 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.245610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-model-cache\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.245763 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.245739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-dshm\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.245822 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.245787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4ssk\" (UniqueName: \"kubernetes.io/projected/41cc10dc-2174-4e66-a69c-be33e0fc9519-kube-api-access-j4ssk\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.245878 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.245850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc10dc-2174-4e66-a69c-be33e0fc9519-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.245968 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.245952    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-home\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.246033 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.246002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.246183 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.246143    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-model-cache\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.248091 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.248056    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-dshm\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.248431 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.248414    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc10dc-2174-4e66-a69c-be33e0fc9519-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.254483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.254459    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4ssk\" (UniqueName: \"kubernetes.io/projected/41cc10dc-2174-4e66-a69c-be33e0fc9519-kube-api-access-j4ssk\") pod \"decom-v1-kserve-6cb797fb75-svcqd\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.404597 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.404568    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:38.682816 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.682792    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"]
Sep 16 13:22:38.684055 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:22:38.684025    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cc10dc_2174_4e66_a69c_be33e0fc9519.slice/crio-77fc0147b825532fc6b4b05955b04aee65e2d956dfa53dcfa39f164ac18e7839 WatchSource:0}: Error finding container 77fc0147b825532fc6b4b05955b04aee65e2d956dfa53dcfa39f164ac18e7839: Status 404 returned error can't find the container with id 77fc0147b825532fc6b4b05955b04aee65e2d956dfa53dcfa39f164ac18e7839
Sep 16 13:22:38.932042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.931951    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd" event={"ID":"41cc10dc-2174-4e66-a69c-be33e0fc9519","Type":"ContainerStarted","Data":"e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58"}
Sep 16 13:22:38.932042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.932018    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd" event={"ID":"41cc10dc-2174-4e66-a69c-be33e0fc9519","Type":"ContainerStarted","Data":"77fc0147b825532fc6b4b05955b04aee65e2d956dfa53dcfa39f164ac18e7839"}
Sep 16 13:22:38.951625 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:38.951578    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd" podStartSLOduration=0.95156401 podStartE2EDuration="951.56401ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:38.949906126 +0000 UTC m=+2080.131048780" watchObservedRunningTime="2026-09-16 13:22:38.95156401 +0000 UTC m=+2080.132706687"
Sep 16 13:22:39.405377 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:39.405344    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:41.582348 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.582308    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"]
Sep 16 13:22:41.585780 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.585758    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.588474 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.588455    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 16 13:22:41.596114 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.596090    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"]
Sep 16 13:22:41.675860 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.675826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7pgg\" (UniqueName: \"kubernetes.io/projected/2ca48c92-488c-4fc6-90b0-044e7f228a07-kube-api-access-m7pgg\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.676015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.675865    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-tmp-dir\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.676015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.675888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-model-cache\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.676015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.675929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-home\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.676015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.675953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca48c92-488c-4fc6-90b0-044e7f228a07-tls-certs\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.676015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.676011    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-dshm\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.776805 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.776776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-dshm\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.776979 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.776847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pgg\" (UniqueName: \"kubernetes.io/projected/2ca48c92-488c-4fc6-90b0-044e7f228a07-kube-api-access-m7pgg\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.776979 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.776869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-tmp-dir\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.776979 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.776892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-model-cache\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.776979 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.776920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-home\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.776979 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.776937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca48c92-488c-4fc6-90b0-044e7f228a07-tls-certs\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.777351 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.777318    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-tmp-dir\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.777422 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.777347    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-model-cache\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.777422 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.777386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-home\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.778984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.778951    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:22:41.779457 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.779433    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-dshm\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.779971 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.779949    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca48c92-488c-4fc6-90b0-044e7f228a07-tls-certs\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.785533 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.785513    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pgg\" (UniqueName: \"kubernetes.io/projected/2ca48c92-488c-4fc6-90b0-044e7f228a07-kube-api-access-m7pgg\") pod \"decom-v2-kserve-77f867d787-zxcfm\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:41.896845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:41.896812    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:42.171921 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:42.171893    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"]
Sep 16 13:22:42.173439 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:22:42.173407    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca48c92_488c_4fc6_90b0_044e7f228a07.slice/crio-e7405c52ef65b6b017150453171032100b29a131dc47402f663b64370864da7e WatchSource:0}: Error finding container e7405c52ef65b6b017150453171032100b29a131dc47402f663b64370864da7e: Status 404 returned error can't find the container with id e7405c52ef65b6b017150453171032100b29a131dc47402f663b64370864da7e
Sep 16 13:22:42.955835 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:42.955800    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm" event={"ID":"2ca48c92-488c-4fc6-90b0-044e7f228a07","Type":"ContainerStarted","Data":"812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc"}
Sep 16 13:22:42.956293 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:42.955854    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm" event={"ID":"2ca48c92-488c-4fc6-90b0-044e7f228a07","Type":"ContainerStarted","Data":"e7405c52ef65b6b017150453171032100b29a131dc47402f663b64370864da7e"}
Sep 16 13:22:42.977952 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:42.977903    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm" podStartSLOduration=1.977889592 podStartE2EDuration="1.977889592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:42.975722559 +0000 UTC m=+2084.156865213" watchObservedRunningTime="2026-09-16 13:22:42.977889592 +0000 UTC m=+2084.159032245"
Sep 16 13:22:43.897790 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:43.897754    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:48.405685 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:48.405637    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:48.418030 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:48.418002    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:49.003884 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:49.003850    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:22:51.897442 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:51.897405    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:51.910486 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:51.910463    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:52.019482 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:52.019452    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:22:55.855467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:55.855439    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:22:58.356909 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:22:58.356879    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:23:00.269352 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:00.269316    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:23:04.113736 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:04.113707    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:23:04.214464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:04.214435    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:23:38.813368 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:38.813327    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:23:46.185365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:46.185332    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"]
Sep 16 13:23:46.195846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:46.195816    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"]
Sep 16 13:23:46.197989 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:46.197964    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:23:46.204902 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:23:46.204864    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:23:46.204902 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:23:46.204902 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:23:46.204902 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72" cmd=["/bin/sleep","15"]
Sep 16 13:23:46.205209 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:46.204917    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" message="" err=<
Sep 16 13:23:46.205209 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:23:46.205209 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:23:46.205209 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:23:46.205209 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:23:46.204929    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:23:46.205209 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:23:46.205209 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:23:46.205209 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" podUID="5952519d-4a0c-4886-9009-1608cc96d915" containerName="main" containerID="cri-o://d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72"
Sep 16 13:23:46.205209 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:46.204966    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" podUID="5952519d-4a0c-4886-9009-1608cc96d915" containerName="main" containerID="cri-o://d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72" gracePeriod=60
Sep 16 13:23:46.262408 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:46.262377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:23:46.963786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:46.963766    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:23:47.097744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.097662    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-tmp-dir\") pod \"5952519d-4a0c-4886-9009-1608cc96d915\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") "
Sep 16 13:23:47.097744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.097703    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-kserve-provision-location\") pod \"5952519d-4a0c-4886-9009-1608cc96d915\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") "
Sep 16 13:23:47.097744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.097736    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-dshm\") pod \"5952519d-4a0c-4886-9009-1608cc96d915\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") "
Sep 16 13:23:47.098010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.097750    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-model-cache\") pod \"5952519d-4a0c-4886-9009-1608cc96d915\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") "
Sep 16 13:23:47.098010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.097767    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5952519d-4a0c-4886-9009-1608cc96d915-cabundle-cert\") pod \"5952519d-4a0c-4886-9009-1608cc96d915\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") "
Sep 16 13:23:47.098010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.097810    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-home\") pod \"5952519d-4a0c-4886-9009-1608cc96d915\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") "
Sep 16 13:23:47.098010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.097851    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpncw\" (UniqueName: \"kubernetes.io/projected/5952519d-4a0c-4886-9009-1608cc96d915-kube-api-access-mpncw\") pod \"5952519d-4a0c-4886-9009-1608cc96d915\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") "
Sep 16 13:23:47.098010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.097879    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5952519d-4a0c-4886-9009-1608cc96d915-tls-certs\") pod \"5952519d-4a0c-4886-9009-1608cc96d915\" (UID: \"5952519d-4a0c-4886-9009-1608cc96d915\") "
Sep 16 13:23:47.098010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.097936    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5952519d-4a0c-4886-9009-1608cc96d915" (UID: "5952519d-4a0c-4886-9009-1608cc96d915"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.098326 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.098027    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-model-cache" (OuterVolumeSpecName: "model-cache") pod "5952519d-4a0c-4886-9009-1608cc96d915" (UID: "5952519d-4a0c-4886-9009-1608cc96d915"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.098326 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.098098    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5952519d-4a0c-4886-9009-1608cc96d915-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5952519d-4a0c-4886-9009-1608cc96d915" (UID: "5952519d-4a0c-4886-9009-1608cc96d915"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:23:47.098326 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.098130    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-home" (OuterVolumeSpecName: "home") pod "5952519d-4a0c-4886-9009-1608cc96d915" (UID: "5952519d-4a0c-4886-9009-1608cc96d915"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.098326 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.098180    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.098326 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.098199    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.098326 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.098214    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5952519d-4a0c-4886-9009-1608cc96d915-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.100185 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.100161    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-dshm" (OuterVolumeSpecName: "dshm") pod "5952519d-4a0c-4886-9009-1608cc96d915" (UID: "5952519d-4a0c-4886-9009-1608cc96d915"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.100300 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.100235    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5952519d-4a0c-4886-9009-1608cc96d915-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5952519d-4a0c-4886-9009-1608cc96d915" (UID: "5952519d-4a0c-4886-9009-1608cc96d915"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:23:47.100531 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.100499    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5952519d-4a0c-4886-9009-1608cc96d915-kube-api-access-mpncw" (OuterVolumeSpecName: "kube-api-access-mpncw") pod "5952519d-4a0c-4886-9009-1608cc96d915" (UID: "5952519d-4a0c-4886-9009-1608cc96d915"). InnerVolumeSpecName "kube-api-access-mpncw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:23:47.119075 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.119053    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5952519d-4a0c-4886-9009-1608cc96d915" (UID: "5952519d-4a0c-4886-9009-1608cc96d915"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:23:47.198969 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.198942    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.198969 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.198965    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.199436 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.198980    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpncw\" (UniqueName: \"kubernetes.io/projected/5952519d-4a0c-4886-9009-1608cc96d915-kube-api-access-mpncw\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.199436 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.198993    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5952519d-4a0c-4886-9009-1608cc96d915-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.199436 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.199005    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5952519d-4a0c-4886-9009-1608cc96d915-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:23:47.315970 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.315937    2575 generic.go:356] "Generic (PLEG): container finished" podID="5952519d-4a0c-4886-9009-1608cc96d915" containerID="d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72" exitCode=0
Sep 16 13:23:47.316101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.315999    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"
Sep 16 13:23:47.316101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.316015    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" event={"ID":"5952519d-4a0c-4886-9009-1608cc96d915","Type":"ContainerDied","Data":"d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72"}
Sep 16 13:23:47.316101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.316066    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" event={"ID":"5952519d-4a0c-4886-9009-1608cc96d915","Type":"ContainerDied","Data":"18e9d95dda77aa40cd0373165d4bfe2064217d1698e8d6d2cd77873e3401ce81"}
Sep 16 13:23:47.316101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.316093    2575 scope.go:117] "RemoveContainer" containerID="d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72"
Sep 16 13:23:47.326824 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.326806    2575 scope.go:117] "RemoveContainer" containerID="d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70"
Sep 16 13:23:47.336311 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.336293    2575 scope.go:117] "RemoveContainer" containerID="d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72"
Sep 16 13:23:47.336567 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:23:47.336549    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72\": container with ID starting with d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72 not found: ID does not exist" containerID="d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72"
Sep 16 13:23:47.336660 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.336579    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72"} err="failed to get container status \"d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72\": rpc error: code = NotFound desc = could not find container \"d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72\": container with ID starting with d39635d7fe46c80ef7a3fe827cd156684941730d026796491514f5a1ba4b2e72 not found: ID does not exist"
Sep 16 13:23:47.336660 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.336604    2575 scope.go:117] "RemoveContainer" containerID="d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70"
Sep 16 13:23:47.336874 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:23:47.336857    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70\": container with ID starting with d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70 not found: ID does not exist" containerID="d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70"
Sep 16 13:23:47.336922 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.336881    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70"} err="failed to get container status \"d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70\": rpc error: code = NotFound desc = could not find container \"d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70\": container with ID starting with d86c30e1c69c345f52114b8e94055e8420bc8d354b3d5721080376ec9b482a70 not found: ID does not exist"
Sep 16 13:23:47.344306 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.344288    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"]
Sep 16 13:23:47.351513 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.351465    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk"]
Sep 16 13:23:47.790378 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.790348    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"]
Sep 16 13:23:47.816323 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.816298    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:23:47.894103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:47.894060    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6f8bb8bf46-k7mnk" podUID="5952519d-4a0c-4886-9009-1608cc96d915" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 13:23:49.329467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:49.329435    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5952519d-4a0c-4886-9009-1608cc96d915" path="/var/lib/kubelet/pods/5952519d-4a0c-4886-9009-1608cc96d915/volumes"
Sep 16 13:23:55.855919 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:23:55.855882    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" podUID="2264f664-ddd7-4124-9743-557d6b99c8f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:24:01.208836 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:01.208778    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" podUID="2264f664-ddd7-4124-9743-557d6b99c8f3" containerName="main" containerID="cri-o://fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9" gracePeriod=45
Sep 16 13:24:02.815168 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:02.815128    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm" podUID="2ca48c92-488c-4fc6-90b0-044e7f228a07" containerName="main" containerID="cri-o://812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc" gracePeriod=45
Sep 16 13:24:03.063574 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.063553    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:24:03.227799 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.227770    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-home\") pod \"2ca48c92-488c-4fc6-90b0-044e7f228a07\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") "
Sep 16 13:24:03.227946 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.227808    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-dshm\") pod \"2ca48c92-488c-4fc6-90b0-044e7f228a07\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") "
Sep 16 13:24:03.227946 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.227856    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca48c92-488c-4fc6-90b0-044e7f228a07-tls-certs\") pod \"2ca48c92-488c-4fc6-90b0-044e7f228a07\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") "
Sep 16 13:24:03.227946 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.227879    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-model-cache\") pod \"2ca48c92-488c-4fc6-90b0-044e7f228a07\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") "
Sep 16 13:24:03.227946 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.227899    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7pgg\" (UniqueName: \"kubernetes.io/projected/2ca48c92-488c-4fc6-90b0-044e7f228a07-kube-api-access-m7pgg\") pod \"2ca48c92-488c-4fc6-90b0-044e7f228a07\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") "
Sep 16 13:24:03.227946 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.227926    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-tmp-dir\") pod \"2ca48c92-488c-4fc6-90b0-044e7f228a07\" (UID: \"2ca48c92-488c-4fc6-90b0-044e7f228a07\") "
Sep 16 13:24:03.228219 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.228034    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-home" (OuterVolumeSpecName: "home") pod "2ca48c92-488c-4fc6-90b0-044e7f228a07" (UID: "2ca48c92-488c-4fc6-90b0-044e7f228a07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:03.228219 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.228163    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ca48c92-488c-4fc6-90b0-044e7f228a07" (UID: "2ca48c92-488c-4fc6-90b0-044e7f228a07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:03.228219 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.228207    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:03.228371 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.228229    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ca48c92-488c-4fc6-90b0-044e7f228a07" (UID: "2ca48c92-488c-4fc6-90b0-044e7f228a07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:03.230204 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.230183    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca48c92-488c-4fc6-90b0-044e7f228a07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ca48c92-488c-4fc6-90b0-044e7f228a07" (UID: "2ca48c92-488c-4fc6-90b0-044e7f228a07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:24:03.230287 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.230238    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca48c92-488c-4fc6-90b0-044e7f228a07-kube-api-access-m7pgg" (OuterVolumeSpecName: "kube-api-access-m7pgg") pod "2ca48c92-488c-4fc6-90b0-044e7f228a07" (UID: "2ca48c92-488c-4fc6-90b0-044e7f228a07"). InnerVolumeSpecName "kube-api-access-m7pgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:24:03.230287 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.230268    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-dshm" (OuterVolumeSpecName: "dshm") pod "2ca48c92-488c-4fc6-90b0-044e7f228a07" (UID: "2ca48c92-488c-4fc6-90b0-044e7f228a07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:03.329074 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.329044    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:03.329190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.329077    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca48c92-488c-4fc6-90b0-044e7f228a07-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:03.329190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.329093    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:03.329190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.329107    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7pgg\" (UniqueName: \"kubernetes.io/projected/2ca48c92-488c-4fc6-90b0-044e7f228a07-kube-api-access-m7pgg\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:03.329190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.329120    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca48c92-488c-4fc6-90b0-044e7f228a07-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:03.407898 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.407871    2575 generic.go:356] "Generic (PLEG): container finished" podID="2ca48c92-488c-4fc6-90b0-044e7f228a07" containerID="812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc" exitCode=0
Sep 16 13:24:03.407994 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.407916    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"
Sep 16 13:24:03.408051 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.407906    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm" event={"ID":"2ca48c92-488c-4fc6-90b0-044e7f228a07","Type":"ContainerDied","Data":"812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc"}
Sep 16 13:24:03.408116 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.408049    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm" event={"ID":"2ca48c92-488c-4fc6-90b0-044e7f228a07","Type":"ContainerDied","Data":"e7405c52ef65b6b017150453171032100b29a131dc47402f663b64370864da7e"}
Sep 16 13:24:03.408116 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.408085    2575 scope.go:117] "RemoveContainer" containerID="812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc"
Sep 16 13:24:03.422464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.422449    2575 scope.go:117] "RemoveContainer" containerID="812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc"
Sep 16 13:24:03.422758 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:24:03.422738    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc\": container with ID starting with 812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc not found: ID does not exist" containerID="812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc"
Sep 16 13:24:03.422823 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.422772    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc"} err="failed to get container status \"812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc\": rpc error: code = NotFound desc = could not find container \"812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc\": container with ID starting with 812266af0ed82bec3c78d2b1cc4a8c235cff28857f755377ffaef552939930cc not found: ID does not exist"
Sep 16 13:24:03.434877 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.434856    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"]
Sep 16 13:24:03.436976 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.436957    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm"]
Sep 16 13:24:03.910299 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.910272    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:03.985209 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:03.985181    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"]
Sep 16 13:24:04.010200 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:04.010166    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-zxcfm" podUID="2ca48c92-488c-4fc6-90b0-044e7f228a07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 13:24:04.016094 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:04.016051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:04.111519 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:04.111489    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:05.329090 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:05.329053    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ca48c92-488c-4fc6-90b0-044e7f228a07" path="/var/lib/kubelet/pods/2ca48c92-488c-4fc6-90b0-044e7f228a07/volumes"
Sep 16 13:24:05.856058 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:05.856020    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" podUID="2264f664-ddd7-4124-9743-557d6b99c8f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:24:08.985512 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:08.985484    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:13.337638 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:13.337602    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:15.855889 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:15.855856    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" podUID="2264f664-ddd7-4124-9743-557d6b99c8f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:24:15.856270 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:15.855950    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:24:19.029057 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.029019    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd" podUID="41cc10dc-2174-4e66-a69c-be33e0fc9519" containerName="main" containerID="cri-o://e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58" gracePeriod=45
Sep 16 13:24:19.281181 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.281127    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:24:19.461677 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.461631    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-home\") pod \"41cc10dc-2174-4e66-a69c-be33e0fc9519\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") "
Sep 16 13:24:19.461849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.461702    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-model-cache\") pod \"41cc10dc-2174-4e66-a69c-be33e0fc9519\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") "
Sep 16 13:24:19.461849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.461722    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-tmp-dir\") pod \"41cc10dc-2174-4e66-a69c-be33e0fc9519\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") "
Sep 16 13:24:19.461849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.461743    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-dshm\") pod \"41cc10dc-2174-4e66-a69c-be33e0fc9519\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") "
Sep 16 13:24:19.461849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.461765    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4ssk\" (UniqueName: \"kubernetes.io/projected/41cc10dc-2174-4e66-a69c-be33e0fc9519-kube-api-access-j4ssk\") pod \"41cc10dc-2174-4e66-a69c-be33e0fc9519\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") "
Sep 16 13:24:19.461849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.461823    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc10dc-2174-4e66-a69c-be33e0fc9519-tls-certs\") pod \"41cc10dc-2174-4e66-a69c-be33e0fc9519\" (UID: \"41cc10dc-2174-4e66-a69c-be33e0fc9519\") "
Sep 16 13:24:19.462122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.461893    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-home" (OuterVolumeSpecName: "home") pod "41cc10dc-2174-4e66-a69c-be33e0fc9519" (UID: "41cc10dc-2174-4e66-a69c-be33e0fc9519"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:19.462122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.461995    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41cc10dc-2174-4e66-a69c-be33e0fc9519" (UID: "41cc10dc-2174-4e66-a69c-be33e0fc9519"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:19.462122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.462007    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-model-cache" (OuterVolumeSpecName: "model-cache") pod "41cc10dc-2174-4e66-a69c-be33e0fc9519" (UID: "41cc10dc-2174-4e66-a69c-be33e0fc9519"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:19.462122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.462101    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:19.462122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.462120    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:19.462390 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.462132    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:19.464013 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.463994    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-dshm" (OuterVolumeSpecName: "dshm") pod "41cc10dc-2174-4e66-a69c-be33e0fc9519" (UID: "41cc10dc-2174-4e66-a69c-be33e0fc9519"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:24:19.464575 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.464555    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41cc10dc-2174-4e66-a69c-be33e0fc9519-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41cc10dc-2174-4e66-a69c-be33e0fc9519" (UID: "41cc10dc-2174-4e66-a69c-be33e0fc9519"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:24:19.464626 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.464572    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cc10dc-2174-4e66-a69c-be33e0fc9519-kube-api-access-j4ssk" (OuterVolumeSpecName: "kube-api-access-j4ssk") pod "41cc10dc-2174-4e66-a69c-be33e0fc9519" (UID: "41cc10dc-2174-4e66-a69c-be33e0fc9519"). InnerVolumeSpecName "kube-api-access-j4ssk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:24:19.489351 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.489326    2575 generic.go:356] "Generic (PLEG): container finished" podID="41cc10dc-2174-4e66-a69c-be33e0fc9519" containerID="e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58" exitCode=0
Sep 16 13:24:19.489462 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.489362    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd" event={"ID":"41cc10dc-2174-4e66-a69c-be33e0fc9519","Type":"ContainerDied","Data":"e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58"}
Sep 16 13:24:19.489462 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.489374    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"
Sep 16 13:24:19.489462 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.489396    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd" event={"ID":"41cc10dc-2174-4e66-a69c-be33e0fc9519","Type":"ContainerDied","Data":"77fc0147b825532fc6b4b05955b04aee65e2d956dfa53dcfa39f164ac18e7839"}
Sep 16 13:24:19.489462 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.489422    2575 scope.go:117] "RemoveContainer" containerID="e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58"
Sep 16 13:24:19.500555 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.500538    2575 scope.go:117] "RemoveContainer" containerID="e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58"
Sep 16 13:24:19.500875 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:24:19.500857    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58\": container with ID starting with e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58 not found: ID does not exist" containerID="e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58"
Sep 16 13:24:19.500955 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.500895    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58"} err="failed to get container status \"e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58\": rpc error: code = NotFound desc = could not find container \"e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58\": container with ID starting with e88774c4b66fcc468e9153935e7d107e81a0b83aa5d4dec6580ef53cf93aad58 not found: ID does not exist"
Sep 16 13:24:19.514335 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.514314    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"]
Sep 16 13:24:19.518622 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.518600    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-svcqd"]
Sep 16 13:24:19.563444 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.563393    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41cc10dc-2174-4e66-a69c-be33e0fc9519-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:19.563444 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.563414    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4ssk\" (UniqueName: \"kubernetes.io/projected/41cc10dc-2174-4e66-a69c-be33e0fc9519-kube-api-access-j4ssk\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:19.563444 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:19.563429    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41cc10dc-2174-4e66-a69c-be33e0fc9519-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:21.329637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:21.329605    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41cc10dc-2174-4e66-a69c-be33e0fc9519" path="/var/lib/kubelet/pods/41cc10dc-2174-4e66-a69c-be33e0fc9519/volumes"
Sep 16 13:24:21.993513 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:21.993485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:25.855529 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:25.855495    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" podUID="2264f664-ddd7-4124-9743-557d6b99c8f3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:24:31.445944 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.445918    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:24:31.550432 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.550358    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2264f664-ddd7-4124-9743-557d6b99c8f3-tls-certs\") pod \"2264f664-ddd7-4124-9743-557d6b99c8f3\" (UID: \"2264f664-ddd7-4124-9743-557d6b99c8f3\") "
Sep 16 13:24:31.550572 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.550454    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wpw6\" (UniqueName: \"kubernetes.io/projected/2264f664-ddd7-4124-9743-557d6b99c8f3-kube-api-access-6wpw6\") pod \"2264f664-ddd7-4124-9743-557d6b99c8f3\" (UID: \"2264f664-ddd7-4124-9743-557d6b99c8f3\") "
Sep 16 13:24:31.552732 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.552706    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2264f664-ddd7-4124-9743-557d6b99c8f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2264f664-ddd7-4124-9743-557d6b99c8f3" (UID: "2264f664-ddd7-4124-9743-557d6b99c8f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:24:31.552872 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.552852    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2264f664-ddd7-4124-9743-557d6b99c8f3-kube-api-access-6wpw6" (OuterVolumeSpecName: "kube-api-access-6wpw6") pod "2264f664-ddd7-4124-9743-557d6b99c8f3" (UID: "2264f664-ddd7-4124-9743-557d6b99c8f3"). InnerVolumeSpecName "kube-api-access-6wpw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:24:31.553266 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.553243    2575 generic.go:356] "Generic (PLEG): container finished" podID="2264f664-ddd7-4124-9743-557d6b99c8f3" containerID="fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9" exitCode=0
Sep 16 13:24:31.553350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.553294    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" event={"ID":"2264f664-ddd7-4124-9743-557d6b99c8f3","Type":"ContainerDied","Data":"fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9"}
Sep 16 13:24:31.553350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.553299    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"
Sep 16 13:24:31.553350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.553336    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf" event={"ID":"2264f664-ddd7-4124-9743-557d6b99c8f3","Type":"ContainerDied","Data":"83a325a5dafa89021d20553994ca1b61d53a3f240998018a339b0bd6fc32c83f"}
Sep 16 13:24:31.553503 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.553362    2575 scope.go:117] "RemoveContainer" containerID="fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9"
Sep 16 13:24:31.568961 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.568942    2575 scope.go:117] "RemoveContainer" containerID="fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9"
Sep 16 13:24:31.569205 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:24:31.569188    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9\": container with ID starting with fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9 not found: ID does not exist" containerID="fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9"
Sep 16 13:24:31.569279 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.569214    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9"} err="failed to get container status \"fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9\": rpc error: code = NotFound desc = could not find container \"fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9\": container with ID starting with fc6b6e249187a16cc1446f0dd0e8527b75c2f34a81501499b26691b5887d33a9 not found: ID does not exist"
Sep 16 13:24:31.585461 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.585434    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"]
Sep 16 13:24:31.593276 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.593249    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69fb4pf"]
Sep 16 13:24:31.651745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.651724    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2264f664-ddd7-4124-9743-557d6b99c8f3-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:31.651745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:31.651745    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wpw6\" (UniqueName: \"kubernetes.io/projected/2264f664-ddd7-4124-9743-557d6b99c8f3-kube-api-access-6wpw6\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:24:33.328922 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:33.328889    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2264f664-ddd7-4124-9743-557d6b99c8f3" path="/var/lib/kubelet/pods/2264f664-ddd7-4124-9743-557d6b99c8f3/volumes"
Sep 16 13:24:36.309403 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:36.309369    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:36.400156 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:36.400128    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:42.667534 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:42.667497    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:43.249210 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:43.249177    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:46.817578 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:46.817545    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:48.097400 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:48.097367    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:48.195884 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:48.195855    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:49.025248 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025214    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"]
Sep 16 13:24:49.025629 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025612    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5952519d-4a0c-4886-9009-1608cc96d915" containerName="storage-initializer"
Sep 16 13:24:49.025744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025632    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5952519d-4a0c-4886-9009-1608cc96d915" containerName="storage-initializer"
Sep 16 13:24:49.025744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025672    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2264f664-ddd7-4124-9743-557d6b99c8f3" containerName="main"
Sep 16 13:24:49.025744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025681    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2264f664-ddd7-4124-9743-557d6b99c8f3" containerName="main"
Sep 16 13:24:49.025744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025704    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5952519d-4a0c-4886-9009-1608cc96d915" containerName="main"
Sep 16 13:24:49.025744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025714    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5952519d-4a0c-4886-9009-1608cc96d915" containerName="main"
Sep 16 13:24:49.025744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025727    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41cc10dc-2174-4e66-a69c-be33e0fc9519" containerName="main"
Sep 16 13:24:49.025744 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025735    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="41cc10dc-2174-4e66-a69c-be33e0fc9519" containerName="main"
Sep 16 13:24:49.026098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025750    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ca48c92-488c-4fc6-90b0-044e7f228a07" containerName="main"
Sep 16 13:24:49.026098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025758    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ca48c92-488c-4fc6-90b0-044e7f228a07" containerName="main"
Sep 16 13:24:49.026098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025850    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="41cc10dc-2174-4e66-a69c-be33e0fc9519" containerName="main"
Sep 16 13:24:49.026098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025860    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2264f664-ddd7-4124-9743-557d6b99c8f3" containerName="main"
Sep 16 13:24:49.026098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025874    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ca48c92-488c-4fc6-90b0-044e7f228a07" containerName="main"
Sep 16 13:24:49.026098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.025883    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5952519d-4a0c-4886-9009-1608cc96d915" containerName="main"
Sep 16 13:24:49.029068 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.029045    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.032604 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.032582    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 13:24:49.032772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.032582    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:24:49.032912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.032897    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 13:24:49.032912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.032906    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 13:24:49.033955 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.033929    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 13:24:49.034018 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.033961    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-jbv4x\""
Sep 16 13:24:49.051299 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.051261    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:49.051544 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.051524    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"]
Sep 16 13:24:49.086089 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.086063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f7c714-8077-47eb-9632-4b1a1942b85e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.086192 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.086090    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.086192 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.086121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.086287 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.086213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-home\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.086287 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.086240    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-dshm\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.086287 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.086261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7c714-8077-47eb-9632-4b1a1942b85e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.086408 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.086293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-model-cache\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.086408 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.086325    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c8fr\" (UniqueName: \"kubernetes.io/projected/d8f7c714-8077-47eb-9632-4b1a1942b85e-kube-api-access-4c8fr\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.187619 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.187587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-model-cache\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.187619 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.187622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c8fr\" (UniqueName: \"kubernetes.io/projected/d8f7c714-8077-47eb-9632-4b1a1942b85e-kube-api-access-4c8fr\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188084 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.187666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f7c714-8077-47eb-9632-4b1a1942b85e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188084 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.187683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188084 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.187705    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188084 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.187752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-home\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188084 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.187774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-dshm\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188084 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.187871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7c714-8077-47eb-9632-4b1a1942b85e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188385 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.188081    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-model-cache\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188385 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.188131    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188385 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.188163    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188385 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.188198    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-home\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.188524 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.188506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7c714-8077-47eb-9632-4b1a1942b85e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.190211 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.190183    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-dshm\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.190312 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.190256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f7c714-8077-47eb-9632-4b1a1942b85e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.197174 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.197150    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c8fr\" (UniqueName: \"kubernetes.io/projected/d8f7c714-8077-47eb-9632-4b1a1942b85e-kube-api-access-4c8fr\") pod \"tokenizer-migration-path-test-kserve-64b5d88575-n4lzx\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.320071 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.320000    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"]
Sep 16 13:24:49.325660 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.325622    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:24:49.328299 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.328275    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-s6v4x\""
Sep 16 13:24:49.332987 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.332964    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:49.338930 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.338736    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:49.341072 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.341051    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"]
Sep 16 13:24:49.389947 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.389912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfrw8\" (UniqueName: \"kubernetes.io/projected/a1108901-0554-4397-94b4-27488a04fcb6-kube-api-access-vfrw8\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh\" (UID: \"a1108901-0554-4397-94b4-27488a04fcb6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:24:49.390116 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.390038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1108901-0554-4397-94b4-27488a04fcb6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh\" (UID: \"a1108901-0554-4397-94b4-27488a04fcb6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:24:49.432861 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.432835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:49.491106 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.491046    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1108901-0554-4397-94b4-27488a04fcb6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh\" (UID: \"a1108901-0554-4397-94b4-27488a04fcb6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:24:49.491388 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.491164    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfrw8\" (UniqueName: \"kubernetes.io/projected/a1108901-0554-4397-94b4-27488a04fcb6-kube-api-access-vfrw8\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh\" (UID: \"a1108901-0554-4397-94b4-27488a04fcb6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:24:49.494038 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.494012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1108901-0554-4397-94b4-27488a04fcb6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh\" (UID: \"a1108901-0554-4397-94b4-27488a04fcb6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:24:49.499622 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.499590    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfrw8\" (UniqueName: \"kubernetes.io/projected/a1108901-0554-4397-94b4-27488a04fcb6-kube-api-access-vfrw8\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh\" (UID: \"a1108901-0554-4397-94b4-27488a04fcb6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:24:49.621004 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.620982    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"]
Sep 16 13:24:49.622320 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:24:49.622295    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f7c714_8077_47eb_9632_4b1a1942b85e.slice/crio-af6732e303768e975fcbf867db5fe5c1e660a7bda102859e43c00cf552955cb4 WatchSource:0}: Error finding container af6732e303768e975fcbf867db5fe5c1e660a7bda102859e43c00cf552955cb4: Status 404 returned error can't find the container with id af6732e303768e975fcbf867db5fe5c1e660a7bda102859e43c00cf552955cb4
Sep 16 13:24:49.638018 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.637998    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:24:49.650372 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.650348    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" event={"ID":"d8f7c714-8077-47eb-9632-4b1a1942b85e","Type":"ContainerStarted","Data":"af6732e303768e975fcbf867db5fe5c1e660a7bda102859e43c00cf552955cb4"}
Sep 16 13:24:49.930314 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:49.930290    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"]
Sep 16 13:24:49.932602 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:24:49.932570    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1108901_0554_4397_94b4_27488a04fcb6.slice/crio-89ad9fc7edec2e3a207ceaab3e61a06194763f4b9404724e7bb40654e8b8ac07 WatchSource:0}: Error finding container 89ad9fc7edec2e3a207ceaab3e61a06194763f4b9404724e7bb40654e8b8ac07: Status 404 returned error can't find the container with id 89ad9fc7edec2e3a207ceaab3e61a06194763f4b9404724e7bb40654e8b8ac07
Sep 16 13:24:50.656767 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:50.656729    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" event={"ID":"d8f7c714-8077-47eb-9632-4b1a1942b85e","Type":"ContainerStarted","Data":"e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8"}
Sep 16 13:24:50.658359 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:50.658333    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" event={"ID":"a1108901-0554-4397-94b4-27488a04fcb6","Type":"ContainerStarted","Data":"d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700"}
Sep 16 13:24:50.658485 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:50.658376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" event={"ID":"a1108901-0554-4397-94b4-27488a04fcb6","Type":"ContainerStarted","Data":"89ad9fc7edec2e3a207ceaab3e61a06194763f4b9404724e7bb40654e8b8ac07"}
Sep 16 13:24:50.658528 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:50.658483    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:24:50.719485 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:50.719438    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" podStartSLOduration=1.719394197 podStartE2EDuration="1.719394197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:50.718179461 +0000 UTC m=+2211.899322151" watchObservedRunningTime="2026-09-16 13:24:50.719394197 +0000 UTC m=+2211.900536851"
Sep 16 13:24:52.674799 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:52.674767    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerID="e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8" exitCode=0
Sep 16 13:24:52.675261 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:52.674825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" event={"ID":"d8f7c714-8077-47eb-9632-4b1a1942b85e","Type":"ContainerDied","Data":"e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8"}
Sep 16 13:24:53.312537 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:53.312501    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:53.682122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:53.682091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" event={"ID":"d8f7c714-8077-47eb-9632-4b1a1942b85e","Type":"ContainerStarted","Data":"b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b"}
Sep 16 13:24:53.740130 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:53.740080    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" podStartSLOduration=3.3518786560000002 podStartE2EDuration="5.740063702s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.388185046s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:24:53.737313676 +0000 UTC m=+2214.918456332" watchObservedRunningTime="2026-09-16 13:24:53.740063702 +0000 UTC m=+2214.921206355"
Sep 16 13:24:54.339226 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:54.339188    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:56.163043 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:56.163014    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:58.881196 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:58.881165    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:24:59.339607 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:59.339512    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:59.352475 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:59.352450    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:24:59.726034 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:24:59.725971    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:25:02.974556 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:02.974521    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:04.057915 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:04.057879    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:04.166754 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:04.166723    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:06.926906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:06.926842    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:07.497799 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:07.497770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:08.050337 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:08.050303    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:11.909041 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:11.909005    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"]
Sep 16 13:25:11.911437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:11.911420    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:11.915895 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:11.915869    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-hhf9n\""
Sep 16 13:25:11.917197 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:11.917171    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 13:25:11.917307 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:11.917246    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 13:25:11.917380 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:11.917185    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 16 13:25:11.929582 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:11.929559    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"]
Sep 16 13:25:11.999980 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:11.999946    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:12.085237 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.085208    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.085400 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.085255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3dc60-8640-49aa-a397-435f1aa50479-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.085400 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.085283    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-dshm\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.085400 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.085374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-home\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.085550 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.085404    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9jh\" (UniqueName: \"kubernetes.io/projected/fbb3dc60-8640-49aa-a397-435f1aa50479-kube-api-access-sr9jh\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.085550 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.085486    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.186902 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.186831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.186902 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.186878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.187152 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.186912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3dc60-8640-49aa-a397-435f1aa50479-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.187152 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.186955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-dshm\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.187271 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.187160    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-home\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.187271 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.187209    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9jh\" (UniqueName: \"kubernetes.io/projected/fbb3dc60-8640-49aa-a397-435f1aa50479-kube-api-access-sr9jh\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.187271 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.187248    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.187416 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.187300    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.187500 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.187482    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-home\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.189407 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.189387    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-dshm\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.189684 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.189666    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3dc60-8640-49aa-a397-435f1aa50479-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.196111 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.196087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9jh\" (UniqueName: \"kubernetes.io/projected/fbb3dc60-8640-49aa-a397-435f1aa50479-kube-api-access-sr9jh\") pod \"tri-v1-kserve-79f8f98bc5-bs2cp\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.222476 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.222452    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:12.512630 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.512607    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"]
Sep 16 13:25:12.514125 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:25:12.514101    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb3dc60_8640_49aa_a397_435f1aa50479.slice/crio-28d403b103fe9aa6cdd60a2d0d7d223c3a53b81f6ce68491a9de81f4743372a1 WatchSource:0}: Error finding container 28d403b103fe9aa6cdd60a2d0d7d223c3a53b81f6ce68491a9de81f4743372a1: Status 404 returned error can't find the container with id 28d403b103fe9aa6cdd60a2d0d7d223c3a53b81f6ce68491a9de81f4743372a1
Sep 16 13:25:12.786329 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.786249    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp" event={"ID":"fbb3dc60-8640-49aa-a397-435f1aa50479","Type":"ContainerStarted","Data":"eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068"}
Sep 16 13:25:12.786329 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.786309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp" event={"ID":"fbb3dc60-8640-49aa-a397-435f1aa50479","Type":"ContainerStarted","Data":"28d403b103fe9aa6cdd60a2d0d7d223c3a53b81f6ce68491a9de81f4743372a1"}
Sep 16 13:25:12.815321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:12.815275    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp" podStartSLOduration=1.815260941 podStartE2EDuration="1.815260941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:25:12.812565624 +0000 UTC m=+2233.993708277" watchObservedRunningTime="2026-09-16 13:25:12.815260941 +0000 UTC m=+2233.996403640"
Sep 16 13:25:13.223448 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:13.223417    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:17.825362 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:17.825324    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:21.669564 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:21.669494    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:25:22.222873 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:22.222840    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:22.235622 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:22.235600    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:22.850132 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:22.850106    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:25:25.455684 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:25.455634    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:45.432694 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:45.432639    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:51.542700 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:51.542624    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:25:51.642802 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:25:51.642775    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:26:18.262958 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:18.262924    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:26:29.102446 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.102404    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:26:29.121279 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.121251    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"]
Sep 16 13:26:29.124066 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.124010    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"]
Sep 16 13:26:29.136429 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:26:29.136396    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:26:29.136429 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:26:29.136429 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:26:29.136429 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b" cmd=["/bin/sleep","15"]
Sep 16 13:26:29.136750 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.136451    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" message="" err=<
Sep 16 13:26:29.136750 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:26:29.136750 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:26:29.136750 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:26:29.136750 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:26:29.136467    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:26:29.136750 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:26:29.136750 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:26:29.136750 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" podUID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerName="main" containerID="cri-o://b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b"
Sep 16 13:26:29.136750 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.136510    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" podUID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerName="main" containerID="cri-o://b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b" gracePeriod=60
Sep 16 13:26:29.171037 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.171005    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:26:29.714313 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.714272    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" podUID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 16 13:26:29.881637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.881614    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:26:29.926939 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.926911    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-kserve-provision-location\") pod \"d8f7c714-8077-47eb-9632-4b1a1942b85e\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") "
Sep 16 13:26:29.927095 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.926964    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f7c714-8077-47eb-9632-4b1a1942b85e-tls-certs\") pod \"d8f7c714-8077-47eb-9632-4b1a1942b85e\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") "
Sep 16 13:26:29.927095 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927012    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-dshm\") pod \"d8f7c714-8077-47eb-9632-4b1a1942b85e\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") "
Sep 16 13:26:29.927095 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927039    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7c714-8077-47eb-9632-4b1a1942b85e-cabundle-cert\") pod \"d8f7c714-8077-47eb-9632-4b1a1942b85e\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") "
Sep 16 13:26:29.927240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927109    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-home\") pod \"d8f7c714-8077-47eb-9632-4b1a1942b85e\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") "
Sep 16 13:26:29.927240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927134    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-tmp-dir\") pod \"d8f7c714-8077-47eb-9632-4b1a1942b85e\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") "
Sep 16 13:26:29.927240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927174    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c8fr\" (UniqueName: \"kubernetes.io/projected/d8f7c714-8077-47eb-9632-4b1a1942b85e-kube-api-access-4c8fr\") pod \"d8f7c714-8077-47eb-9632-4b1a1942b85e\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") "
Sep 16 13:26:29.927240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927212    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-model-cache\") pod \"d8f7c714-8077-47eb-9632-4b1a1942b85e\" (UID: \"d8f7c714-8077-47eb-9632-4b1a1942b85e\") "
Sep 16 13:26:29.927603 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927528    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-home" (OuterVolumeSpecName: "home") pod "d8f7c714-8077-47eb-9632-4b1a1942b85e" (UID: "d8f7c714-8077-47eb-9632-4b1a1942b85e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:29.927603 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927595    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8f7c714-8077-47eb-9632-4b1a1942b85e" (UID: "d8f7c714-8077-47eb-9632-4b1a1942b85e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:29.927865 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927803    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f7c714-8077-47eb-9632-4b1a1942b85e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d8f7c714-8077-47eb-9632-4b1a1942b85e" (UID: "d8f7c714-8077-47eb-9632-4b1a1942b85e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:26:29.927932 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.927888    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8f7c714-8077-47eb-9632-4b1a1942b85e" (UID: "d8f7c714-8077-47eb-9632-4b1a1942b85e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:29.930741 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.930708    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f7c714-8077-47eb-9632-4b1a1942b85e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8f7c714-8077-47eb-9632-4b1a1942b85e" (UID: "d8f7c714-8077-47eb-9632-4b1a1942b85e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:26:29.931479 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.931442    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-dshm" (OuterVolumeSpecName: "dshm") pod "d8f7c714-8077-47eb-9632-4b1a1942b85e" (UID: "d8f7c714-8077-47eb-9632-4b1a1942b85e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:29.931589 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.931549    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f7c714-8077-47eb-9632-4b1a1942b85e-kube-api-access-4c8fr" (OuterVolumeSpecName: "kube-api-access-4c8fr") pod "d8f7c714-8077-47eb-9632-4b1a1942b85e" (UID: "d8f7c714-8077-47eb-9632-4b1a1942b85e"). InnerVolumeSpecName "kube-api-access-4c8fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:26:29.953254 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:29.953231    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d8f7c714-8077-47eb-9632-4b1a1942b85e" (UID: "d8f7c714-8077-47eb-9632-4b1a1942b85e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:26:30.029017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.028950    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.029017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.028978    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.029017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.028992    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c8fr\" (UniqueName: \"kubernetes.io/projected/d8f7c714-8077-47eb-9632-4b1a1942b85e-kube-api-access-4c8fr\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.029017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.029005    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.029223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.029019    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.029223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.029032    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f7c714-8077-47eb-9632-4b1a1942b85e-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.029223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.029043    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f7c714-8077-47eb-9632-4b1a1942b85e-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.029223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.029055    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8f7c714-8077-47eb-9632-4b1a1942b85e-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:26:30.187853 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.187820    2575 generic.go:356] "Generic (PLEG): container finished" podID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerID="b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b" exitCode=0
Sep 16 13:26:30.188316 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.187881    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"
Sep 16 13:26:30.188316 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.187876    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" event={"ID":"d8f7c714-8077-47eb-9632-4b1a1942b85e","Type":"ContainerDied","Data":"b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b"}
Sep 16 13:26:30.188316 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.188010    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx" event={"ID":"d8f7c714-8077-47eb-9632-4b1a1942b85e","Type":"ContainerDied","Data":"af6732e303768e975fcbf867db5fe5c1e660a7bda102859e43c00cf552955cb4"}
Sep 16 13:26:30.188316 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.188032    2575 scope.go:117] "RemoveContainer" containerID="b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b"
Sep 16 13:26:30.200434 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.200417    2575 scope.go:117] "RemoveContainer" containerID="e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8"
Sep 16 13:26:30.208365 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.208349    2575 scope.go:117] "RemoveContainer" containerID="b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b"
Sep 16 13:26:30.208659 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:26:30.208627    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b\": container with ID starting with b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b not found: ID does not exist" containerID="b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b"
Sep 16 13:26:30.208717 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.208669    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b"} err="failed to get container status \"b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b\": rpc error: code = NotFound desc = could not find container \"b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b\": container with ID starting with b2e1f2d57037cb7d440ae1b748596c3c8e0edf94a25a89c7a386b8a54919f23b not found: ID does not exist"
Sep 16 13:26:30.208717 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.208687    2575 scope.go:117] "RemoveContainer" containerID="e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8"
Sep 16 13:26:30.208934 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:26:30.208918    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8\": container with ID starting with e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8 not found: ID does not exist" containerID="e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8"
Sep 16 13:26:30.208979 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.208940    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8"} err="failed to get container status \"e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8\": rpc error: code = NotFound desc = could not find container \"e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8\": container with ID starting with e608f6ef4fb9400d4023ca1cd43f7e83f98044ba2251ae6ddf07436dff2db8b8 not found: ID does not exist"
Sep 16 13:26:30.212702 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.212682    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"]
Sep 16 13:26:30.217100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:30.217081    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-64b5d88575-n4lzx"]
Sep 16 13:26:31.331180 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:31.331143    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8f7c714-8077-47eb-9632-4b1a1942b85e" path="/var/lib/kubelet/pods/d8f7c714-8077-47eb-9632-4b1a1942b85e/volumes"
Sep 16 13:26:31.669816 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:31.669780    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" podUID="a1108901-0554-4397-94b4-27488a04fcb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:26:41.670329 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:41.670284    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" podUID="a1108901-0554-4397-94b4-27488a04fcb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:26:44.147690 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:44.147608    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" podUID="a1108901-0554-4397-94b4-27488a04fcb6" containerName="main" containerID="cri-o://d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700" gracePeriod=45
Sep 16 13:26:51.670016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:51.669978    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" podUID="a1108901-0554-4397-94b4-27488a04fcb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:26:51.670518 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:26:51.670085    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:27:01.670308 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:01.670274    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" podUID="a1108901-0554-4397-94b4-27488a04fcb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:27:09.110299 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:09.110268    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:09.212411 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:09.212377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:09.328078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:09.328036    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:09.420877 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:09.420796    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:09.682802 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:09.682713    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"]
Sep 16 13:27:09.709757 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:09.709729    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:10.501073 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:10.501040    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:11.670109 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:11.670072    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" podUID="a1108901-0554-4397-94b4-27488a04fcb6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:27:14.103877 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.103843    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:14.300222 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.300188    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:14.400606 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.400579    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:27:14.424755 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.424731    2575 generic.go:356] "Generic (PLEG): container finished" podID="a1108901-0554-4397-94b4-27488a04fcb6" containerID="d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700" exitCode=0
Sep 16 13:27:14.424864 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.424767    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" event={"ID":"a1108901-0554-4397-94b4-27488a04fcb6","Type":"ContainerDied","Data":"d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700"}
Sep 16 13:27:14.424864 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.424833    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh" event={"ID":"a1108901-0554-4397-94b4-27488a04fcb6","Type":"ContainerDied","Data":"89ad9fc7edec2e3a207ceaab3e61a06194763f4b9404724e7bb40654e8b8ac07"}
Sep 16 13:27:14.424942 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.424863    2575 scope.go:117] "RemoveContainer" containerID="d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700"
Sep 16 13:27:14.424942 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.424782    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"
Sep 16 13:27:14.433397 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.433380    2575 scope.go:117] "RemoveContainer" containerID="d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700"
Sep 16 13:27:14.433682 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:27:14.433660    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700\": container with ID starting with d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700 not found: ID does not exist" containerID="d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700"
Sep 16 13:27:14.433761 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.433690    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700"} err="failed to get container status \"d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700\": rpc error: code = NotFound desc = could not find container \"d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700\": container with ID starting with d4a9af9ff4b822e410b909a5283c2bb70f9ab84f493c3ed6f40d392c1df52700 not found: ID does not exist"
Sep 16 13:27:14.496664 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.496617    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1108901-0554-4397-94b4-27488a04fcb6-tls-certs\") pod \"a1108901-0554-4397-94b4-27488a04fcb6\" (UID: \"a1108901-0554-4397-94b4-27488a04fcb6\") "
Sep 16 13:27:14.496786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.496696    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfrw8\" (UniqueName: \"kubernetes.io/projected/a1108901-0554-4397-94b4-27488a04fcb6-kube-api-access-vfrw8\") pod \"a1108901-0554-4397-94b4-27488a04fcb6\" (UID: \"a1108901-0554-4397-94b4-27488a04fcb6\") "
Sep 16 13:27:14.498827 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.498798    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1108901-0554-4397-94b4-27488a04fcb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1108901-0554-4397-94b4-27488a04fcb6" (UID: "a1108901-0554-4397-94b4-27488a04fcb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:27:14.498827 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.498803    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1108901-0554-4397-94b4-27488a04fcb6-kube-api-access-vfrw8" (OuterVolumeSpecName: "kube-api-access-vfrw8") pod "a1108901-0554-4397-94b4-27488a04fcb6" (UID: "a1108901-0554-4397-94b4-27488a04fcb6"). InnerVolumeSpecName "kube-api-access-vfrw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:27:14.598017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.597988    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1108901-0554-4397-94b4-27488a04fcb6-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:27:14.598017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.598013    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfrw8\" (UniqueName: \"kubernetes.io/projected/a1108901-0554-4397-94b4-27488a04fcb6-kube-api-access-vfrw8\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:27:14.749934 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.749888    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"]
Sep 16 13:27:14.756797 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:14.756774    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785pzvjh"]
Sep 16 13:27:15.328893 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:15.328855    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1108901-0554-4397-94b4-27488a04fcb6" path="/var/lib/kubelet/pods/a1108901-0554-4397-94b4-27488a04fcb6/volumes"
Sep 16 13:27:19.109810 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:19.109771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:19.210091 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:19.210061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:24.704288 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.704228    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp" podUID="fbb3dc60-8640-49aa-a397-435f1aa50479" containerName="main" containerID="cri-o://eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068" gracePeriod=45
Sep 16 13:27:24.839827 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.839793    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp" podUID="fbb3dc60-8640-49aa-a397-435f1aa50479" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 16 13:27:24.951841 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.951816    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:27:24.980684 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.980603    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr9jh\" (UniqueName: \"kubernetes.io/projected/fbb3dc60-8640-49aa-a397-435f1aa50479-kube-api-access-sr9jh\") pod \"fbb3dc60-8640-49aa-a397-435f1aa50479\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") "
Sep 16 13:27:24.980684 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.980663    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-home\") pod \"fbb3dc60-8640-49aa-a397-435f1aa50479\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") "
Sep 16 13:27:24.980845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.980727    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-dshm\") pod \"fbb3dc60-8640-49aa-a397-435f1aa50479\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") "
Sep 16 13:27:24.980845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.980783    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-tmp-dir\") pod \"fbb3dc60-8640-49aa-a397-435f1aa50479\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") "
Sep 16 13:27:24.980845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.980826    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-model-cache\") pod \"fbb3dc60-8640-49aa-a397-435f1aa50479\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") "
Sep 16 13:27:24.981000 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.980861    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3dc60-8640-49aa-a397-435f1aa50479-tls-certs\") pod \"fbb3dc60-8640-49aa-a397-435f1aa50479\" (UID: \"fbb3dc60-8640-49aa-a397-435f1aa50479\") "
Sep 16 13:27:24.981000 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.980922    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-home" (OuterVolumeSpecName: "home") pod "fbb3dc60-8640-49aa-a397-435f1aa50479" (UID: "fbb3dc60-8640-49aa-a397-435f1aa50479"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:24.981100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.981021    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbb3dc60-8640-49aa-a397-435f1aa50479" (UID: "fbb3dc60-8640-49aa-a397-435f1aa50479"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:24.981152 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.981100    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:27:24.981152 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.981105    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbb3dc60-8640-49aa-a397-435f1aa50479" (UID: "fbb3dc60-8640-49aa-a397-435f1aa50479"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:24.981152 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.981117    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:27:24.983021 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.982999    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-dshm" (OuterVolumeSpecName: "dshm") pod "fbb3dc60-8640-49aa-a397-435f1aa50479" (UID: "fbb3dc60-8640-49aa-a397-435f1aa50479"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:27:24.983105 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.982999    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb3dc60-8640-49aa-a397-435f1aa50479-kube-api-access-sr9jh" (OuterVolumeSpecName: "kube-api-access-sr9jh") pod "fbb3dc60-8640-49aa-a397-435f1aa50479" (UID: "fbb3dc60-8640-49aa-a397-435f1aa50479"). InnerVolumeSpecName "kube-api-access-sr9jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:27:24.983105 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:24.983049    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb3dc60-8640-49aa-a397-435f1aa50479-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbb3dc60-8640-49aa-a397-435f1aa50479" (UID: "fbb3dc60-8640-49aa-a397-435f1aa50479"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:27:25.081688 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.081638    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:27:25.081688 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.081684    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb3dc60-8640-49aa-a397-435f1aa50479-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:27:25.081688 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.081693    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr9jh\" (UniqueName: \"kubernetes.io/projected/fbb3dc60-8640-49aa-a397-435f1aa50479-kube-api-access-sr9jh\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:27:25.081899 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.081703    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb3dc60-8640-49aa-a397-435f1aa50479-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:27:25.483039 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.483009    2575 generic.go:356] "Generic (PLEG): container finished" podID="fbb3dc60-8640-49aa-a397-435f1aa50479" containerID="eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068" exitCode=0
Sep 16 13:27:25.483225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.483055    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"
Sep 16 13:27:25.483225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.483095    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp" event={"ID":"fbb3dc60-8640-49aa-a397-435f1aa50479","Type":"ContainerDied","Data":"eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068"}
Sep 16 13:27:25.483225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.483155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp" event={"ID":"fbb3dc60-8640-49aa-a397-435f1aa50479","Type":"ContainerDied","Data":"28d403b103fe9aa6cdd60a2d0d7d223c3a53b81f6ce68491a9de81f4743372a1"}
Sep 16 13:27:25.483225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.483187    2575 scope.go:117] "RemoveContainer" containerID="eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068"
Sep 16 13:27:25.491772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.491752    2575 scope.go:117] "RemoveContainer" containerID="eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068"
Sep 16 13:27:25.492079 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:27:25.492060    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068\": container with ID starting with eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068 not found: ID does not exist" containerID="eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068"
Sep 16 13:27:25.492153 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.492100    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068"} err="failed to get container status \"eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068\": rpc error: code = NotFound desc = could not find container \"eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068\": container with ID starting with eda52bd0ddeb211bd18d5e739c200a53aff95f8c2dbab0adb95f22db89ac7068 not found: ID does not exist"
Sep 16 13:27:25.504047 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.504028    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"]
Sep 16 13:27:25.505991 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:25.505973    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bs2cp"]
Sep 16 13:27:26.421381 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:26.421353    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:27.328533 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:27.328498    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb3dc60-8640-49aa-a397-435f1aa50479" path="/var/lib/kubelet/pods/fbb3dc60-8640-49aa-a397-435f1aa50479/volumes"
Sep 16 13:27:32.522868 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:32.522827    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:32.591118 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:32.591093    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:49.025682 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.025631    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"]
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026000    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerName="storage-initializer"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026011    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerName="storage-initializer"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026026    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1108901-0554-4397-94b4-27488a04fcb6" containerName="main"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026031    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1108901-0554-4397-94b4-27488a04fcb6" containerName="main"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026041    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerName="main"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026046    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerName="main"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026065    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb3dc60-8640-49aa-a397-435f1aa50479" containerName="main"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026070    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb3dc60-8640-49aa-a397-435f1aa50479" containerName="main"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026121    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb3dc60-8640-49aa-a397-435f1aa50479" containerName="main"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026128    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f7c714-8077-47eb-9632-4b1a1942b85e" containerName="main"
Sep 16 13:27:49.026143 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.026136    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1108901-0554-4397-94b4-27488a04fcb6" containerName="main"
Sep 16 13:27:49.029408 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.029387    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.032996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.032976    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 13:27:49.033294 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.033268    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 13:27:49.033294 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.033292    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:27:49.034414 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.034392    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 13:27:49.034514 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.034424    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-d4hkr\""
Sep 16 13:27:49.034514 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.034429    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 13:27:49.063887 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.063862    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"]
Sep 16 13:27:49.072577 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.072558    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:49.167388 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.167364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-model-cache\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.167522 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.167392    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.167522 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.167414    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.167522 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.167440    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tls-certs\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.167522 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.167471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-home\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.167522 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.167513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-dshm\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.167734 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.167529    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.167734 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.167558    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxl7\" (UniqueName: \"kubernetes.io/projected/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kube-api-access-twxl7\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.268506 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.268475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tls-certs\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.268637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.268509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-home\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.268637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.268555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-dshm\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.268637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.268579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.268637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.268628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxl7\" (UniqueName: \"kubernetes.io/projected/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kube-api-access-twxl7\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.268901 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.268701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-model-cache\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.268901 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.268725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.268901 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.268762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.269064 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.269043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.269125 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.269059    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.269184 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.269122    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-model-cache\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.269244 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.269226    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-home\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.269456 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.269435    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.270909 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.270891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-dshm\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.271183 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.271164    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tls-certs\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.278283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.278233    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxl7\" (UniqueName: \"kubernetes.io/projected/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kube-api-access-twxl7\") pod \"tokenizer-template-args-test-kserve-5f9bf4f597-vrz72\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.340000 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.339977    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:49.380868 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.380818    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"]
Sep 16 13:27:49.385484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.385404    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:27:49.388602 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.388505    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-mzq8x\""
Sep 16 13:27:49.390154 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.390091    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:49.395694 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.395630    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"]
Sep 16 13:27:49.470501 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.470467    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lll\" (UniqueName: \"kubernetes.io/projected/fa949965-69f8-4118-942e-df4a9d38e31a-kube-api-access-p8lll\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl\" (UID: \"fa949965-69f8-4118-942e-df4a9d38e31a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:27:49.470670 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.470586    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949965-69f8-4118-942e-df4a9d38e31a-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl\" (UID: \"fa949965-69f8-4118-942e-df4a9d38e31a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:27:49.482380 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.482339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:49.571768 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.571697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949965-69f8-4118-942e-df4a9d38e31a-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl\" (UID: \"fa949965-69f8-4118-942e-df4a9d38e31a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:27:49.571878 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.571780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lll\" (UniqueName: \"kubernetes.io/projected/fa949965-69f8-4118-942e-df4a9d38e31a-kube-api-access-p8lll\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl\" (UID: \"fa949965-69f8-4118-942e-df4a9d38e31a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:27:49.574338 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.574315    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949965-69f8-4118-942e-df4a9d38e31a-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl\" (UID: \"fa949965-69f8-4118-942e-df4a9d38e31a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:27:49.581495 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.581476    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lll\" (UniqueName: \"kubernetes.io/projected/fa949965-69f8-4118-942e-df4a9d38e31a-kube-api-access-p8lll\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl\" (UID: \"fa949965-69f8-4118-942e-df4a9d38e31a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:27:49.626494 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.626461    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"]
Sep 16 13:27:49.627013 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:27:49.626992    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f6d3f4_9865_4675_a192_b7b3cb4a3b26.slice/crio-b3c4c4f5e4c1225945a18a62fe4a48eb813768a41f5ce5d4ef13125c4d4ba1d4 WatchSource:0}: Error finding container b3c4c4f5e4c1225945a18a62fe4a48eb813768a41f5ce5d4ef13125c4d4ba1d4: Status 404 returned error can't find the container with id b3c4c4f5e4c1225945a18a62fe4a48eb813768a41f5ce5d4ef13125c4d4ba1d4
Sep 16 13:27:49.699824 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.699788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:27:49.980909 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:49.980884    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"]
Sep 16 13:27:49.982715 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:27:49.982687    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa949965_69f8_4118_942e_df4a9d38e31a.slice/crio-67525bf52c7dd654e2435bc455340e65f3db01b805fde0cf9ff1089bc7f99411 WatchSource:0}: Error finding container 67525bf52c7dd654e2435bc455340e65f3db01b805fde0cf9ff1089bc7f99411: Status 404 returned error can't find the container with id 67525bf52c7dd654e2435bc455340e65f3db01b805fde0cf9ff1089bc7f99411
Sep 16 13:27:50.610151 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:50.610116    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" event={"ID":"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26","Type":"ContainerStarted","Data":"19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7"}
Sep 16 13:27:50.610628 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:50.610186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" event={"ID":"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26","Type":"ContainerStarted","Data":"b3c4c4f5e4c1225945a18a62fe4a48eb813768a41f5ce5d4ef13125c4d4ba1d4"}
Sep 16 13:27:50.611857 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:50.611823    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" event={"ID":"fa949965-69f8-4118-942e-df4a9d38e31a","Type":"ContainerStarted","Data":"b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438"}
Sep 16 13:27:50.611967 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:50.611877    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" event={"ID":"fa949965-69f8-4118-942e-df4a9d38e31a","Type":"ContainerStarted","Data":"67525bf52c7dd654e2435bc455340e65f3db01b805fde0cf9ff1089bc7f99411"}
Sep 16 13:27:50.611967 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:50.611908    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:27:50.652624 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:50.652578    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" podStartSLOduration=1.652563346 podStartE2EDuration="1.652563346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:27:50.649160756 +0000 UTC m=+2391.830303420" watchObservedRunningTime="2026-09-16 13:27:50.652563346 +0000 UTC m=+2391.833706000"
Sep 16 13:27:52.319351 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:52.319322    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:52.623563 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:52.623528    2575 generic.go:356] "Generic (PLEG): container finished" podID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerID="19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7" exitCode=0
Sep 16 13:27:52.623793 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:52.623607    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" event={"ID":"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26","Type":"ContainerDied","Data":"19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7"}
Sep 16 13:27:53.630974 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:53.630942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" event={"ID":"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26","Type":"ContainerStarted","Data":"fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545"}
Sep 16 13:27:53.656766 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:53.656721    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" podStartSLOduration=3.230240439 podStartE2EDuration="5.656707318s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.426466879s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:27:53.653910448 +0000 UTC m=+2394.835053115" watchObservedRunningTime="2026-09-16 13:27:53.656707318 +0000 UTC m=+2394.837849970"
Sep 16 13:27:54.340626 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:54.340592    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:58.190730 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:58.190694    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:58.253043 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:58.253014    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:27:59.341142 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:59.341112    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:59.353941 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:59.353918    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:27:59.676291 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:27:59.676268    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:28:01.065234 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:01.065195    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:28:21.619676 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:21.619572    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:28:27.211126 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:27.211089    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:28:27.311441 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:27.311408    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:28:32.557047 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:32.557010    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:28:42.519564 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.519528    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:28:42.535734 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.535691    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"]
Sep 16 13:28:42.541696 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.541669    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"]
Sep 16 13:28:42.546376 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:28:42.546339    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:28:42.546376 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:28:42.546376 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:28:42.546376 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545" cmd=["/bin/sleep","15"]
Sep 16 13:28:42.546627 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.546389    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" message="" err=<
Sep 16 13:28:42.546627 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:28:42.546627 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:28:42.546627 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:28:42.546627 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:28:42.546403    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:28:42.546627 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:28:42.546627 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:28:42.546627 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" podUID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerName="main" containerID="cri-o://fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545"
Sep 16 13:28:42.546627 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.546450    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" podUID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerName="main" containerID="cri-o://fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545" gracePeriod=60
Sep 16 13:28:42.581017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.580988    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:28:42.662740 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.662701    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" podUID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 13:28:42.801732 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.801710    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:28:42.887680 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.887630    2575 generic.go:356] "Generic (PLEG): container finished" podID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerID="fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545" exitCode=0
Sep 16 13:28:42.887849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.887698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" event={"ID":"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26","Type":"ContainerDied","Data":"fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545"}
Sep 16 13:28:42.887849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.887751    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72" event={"ID":"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26","Type":"ContainerDied","Data":"b3c4c4f5e4c1225945a18a62fe4a48eb813768a41f5ce5d4ef13125c4d4ba1d4"}
Sep 16 13:28:42.887849 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.887703    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"
Sep 16 13:28:42.887983 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.887781    2575 scope.go:117] "RemoveContainer" containerID="fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545"
Sep 16 13:28:42.896899 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.896881    2575 scope.go:117] "RemoveContainer" containerID="19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7"
Sep 16 13:28:42.904720 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.904698    2575 scope.go:117] "RemoveContainer" containerID="fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545"
Sep 16 13:28:42.904985 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:28:42.904968    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545\": container with ID starting with fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545 not found: ID does not exist" containerID="fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545"
Sep 16 13:28:42.905042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.904994    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545"} err="failed to get container status \"fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545\": rpc error: code = NotFound desc = could not find container \"fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545\": container with ID starting with fdeb87ea04845302be8220faa2ad237fd38a42ff3b6f35269271bc4ddad5e545 not found: ID does not exist"
Sep 16 13:28:42.905042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905010    2575 scope.go:117] "RemoveContainer" containerID="19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7"
Sep 16 13:28:42.905272 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:28:42.905254    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7\": container with ID starting with 19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7 not found: ID does not exist" containerID="19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7"
Sep 16 13:28:42.905322 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905277    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7"} err="failed to get container status \"19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7\": rpc error: code = NotFound desc = could not find container \"19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7\": container with ID starting with 19e4ecd695959725dcf9b47f1eff45374cd3fb5392d55b5d6774703ce5a820d7 not found: ID does not exist"
Sep 16 13:28:42.905394 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905379    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-dshm\") pod \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") "
Sep 16 13:28:42.905437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905409    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tmp-dir\") pod \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") "
Sep 16 13:28:42.905437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905428    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tls-certs\") pod \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") "
Sep 16 13:28:42.905506 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905442    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-home\") pod \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") "
Sep 16 13:28:42.905506 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905467    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kserve-provision-location\") pod \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") "
Sep 16 13:28:42.905603 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905529    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-cabundle-cert\") pod \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") "
Sep 16 13:28:42.905603 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905556    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-model-cache\") pod \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") "
Sep 16 13:28:42.905738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905622    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twxl7\" (UniqueName: \"kubernetes.io/projected/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kube-api-access-twxl7\") pod \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\" (UID: \"b4f6d3f4-9865-4675-a192-b7b3cb4a3b26\") "
Sep 16 13:28:42.905738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905689    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" (UID: "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:42.905889 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905863    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-home" (OuterVolumeSpecName: "home") pod "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" (UID: "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:42.905971 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905893    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" (UID: "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:42.905971 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905921    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" (UID: "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:28:42.906072 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.905995    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:28:42.906072 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.906015    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:28:42.907805 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.907782    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-dshm" (OuterVolumeSpecName: "dshm") pod "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" (UID: "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:42.907894 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.907874    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" (UID: "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:28:42.907957 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.907940    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kube-api-access-twxl7" (OuterVolumeSpecName: "kube-api-access-twxl7") pod "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" (UID: "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26"). InnerVolumeSpecName "kube-api-access-twxl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:28:42.926575 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:42.926550    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" (UID: "b4f6d3f4-9865-4675-a192-b7b3cb4a3b26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:28:43.006977 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:43.006956    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.006977 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:43.006975    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxl7\" (UniqueName: \"kubernetes.io/projected/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kube-api-access-twxl7\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.007103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:43.006985    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.007103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:43.006994    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.007103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:43.007003    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.007103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:43.007010    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:28:43.213225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:43.213197    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"]
Sep 16 13:28:43.216886 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:43.216865    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5f9bf4f597-vrz72"]
Sep 16 13:28:43.330115 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:43.330087    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" path="/var/lib/kubelet/pods/b4f6d3f4-9865-4675-a192-b7b3cb4a3b26/volumes"
Sep 16 13:28:51.619524 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:51.619483    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" podUID="fa949965-69f8-4118-942e-df4a9d38e31a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:28:57.562080 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:28:57.562044    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" podUID="fa949965-69f8-4118-942e-df4a9d38e31a" containerName="main" containerID="cri-o://b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438" gracePeriod=45
Sep 16 13:29:01.619052 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:01.619015    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" podUID="fa949965-69f8-4118-942e-df4a9d38e31a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:29:10.573152 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:10.573116    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:10.679199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:10.679165    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:10.759524 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:10.759495    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:10.825161 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:10.825084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:11.102507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:11.102424    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:11.619718 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:11.619680    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" podUID="fa949965-69f8-4118-942e-df4a9d38e31a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:29:11.620163 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:11.619810    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:29:15.646211 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:15.646175    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:21.619121 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:21.619085    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" podUID="fa949965-69f8-4118-942e-df4a9d38e31a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:29:21.951361 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:21.951257    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:22.016933 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:22.016901    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:22.842277 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:22.842245    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:27.817410 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:27.817386    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:29:27.960048 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:27.960020    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8lll\" (UniqueName: \"kubernetes.io/projected/fa949965-69f8-4118-942e-df4a9d38e31a-kube-api-access-p8lll\") pod \"fa949965-69f8-4118-942e-df4a9d38e31a\" (UID: \"fa949965-69f8-4118-942e-df4a9d38e31a\") "
Sep 16 13:29:27.960194 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:27.960095    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949965-69f8-4118-942e-df4a9d38e31a-tls-certs\") pod \"fa949965-69f8-4118-942e-df4a9d38e31a\" (UID: \"fa949965-69f8-4118-942e-df4a9d38e31a\") "
Sep 16 13:29:27.962155 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:27.962126    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa949965-69f8-4118-942e-df4a9d38e31a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa949965-69f8-4118-942e-df4a9d38e31a" (UID: "fa949965-69f8-4118-942e-df4a9d38e31a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:29:27.962273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:27.962180    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa949965-69f8-4118-942e-df4a9d38e31a-kube-api-access-p8lll" (OuterVolumeSpecName: "kube-api-access-p8lll") pod "fa949965-69f8-4118-942e-df4a9d38e31a" (UID: "fa949965-69f8-4118-942e-df4a9d38e31a"). InnerVolumeSpecName "kube-api-access-p8lll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:29:28.060623 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.060594    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949965-69f8-4118-942e-df4a9d38e31a-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:29:28.060623 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.060620    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8lll\" (UniqueName: \"kubernetes.io/projected/fa949965-69f8-4118-942e-df4a9d38e31a-kube-api-access-p8lll\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:29:28.124688 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.124623    2575 generic.go:356] "Generic (PLEG): container finished" podID="fa949965-69f8-4118-942e-df4a9d38e31a" containerID="b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438" exitCode=0
Sep 16 13:29:28.124788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.124687    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"
Sep 16 13:29:28.124788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.124710    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" event={"ID":"fa949965-69f8-4118-942e-df4a9d38e31a","Type":"ContainerDied","Data":"b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438"}
Sep 16 13:29:28.124788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.124784    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl" event={"ID":"fa949965-69f8-4118-942e-df4a9d38e31a","Type":"ContainerDied","Data":"67525bf52c7dd654e2435bc455340e65f3db01b805fde0cf9ff1089bc7f99411"}
Sep 16 13:29:28.124897 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.124807    2575 scope.go:117] "RemoveContainer" containerID="b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438"
Sep 16 13:29:28.134034 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.134018    2575 scope.go:117] "RemoveContainer" containerID="b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438"
Sep 16 13:29:28.134298 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:29:28.134278    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438\": container with ID starting with b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438 not found: ID does not exist" containerID="b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438"
Sep 16 13:29:28.134362 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.134307    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438"} err="failed to get container status \"b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438\": rpc error: code = NotFound desc = could not find container \"b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438\": container with ID starting with b4aa69d98b098bc1f7dea7cf8b2e89a1161d59682b6f033d4e19189629eed438 not found: ID does not exist"
Sep 16 13:29:28.146998 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.146977    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"]
Sep 16 13:29:28.152691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:28.152670    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dtx8wl"]
Sep 16 13:29:29.329656 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:29.329621    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa949965-69f8-4118-942e-df4a9d38e31a" path="/var/lib/kubelet/pods/fa949965-69f8-4118-942e-df4a9d38e31a/volumes"
Sep 16 13:29:30.415911 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:30.415875    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:31.113915 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:31.113884    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:31.214003 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:31.213972    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:35.178241 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:35.178197    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:37.696283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:37.696247    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:43.372540 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:43.372503    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:43.472746 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:43.472716    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:48.604490 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:48.604444    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:54.797155 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:54.797119    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:29:54.899636 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:29:54.899597    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:14.696064 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.696028    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"]
Sep 16 13:30:14.696504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.696361    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerName="storage-initializer"
Sep 16 13:30:14.696504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.696372    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerName="storage-initializer"
Sep 16 13:30:14.696504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.696384    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa949965-69f8-4118-942e-df4a9d38e31a" containerName="main"
Sep 16 13:30:14.696504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.696390    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa949965-69f8-4118-942e-df4a9d38e31a" containerName="main"
Sep 16 13:30:14.696504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.696410    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerName="main"
Sep 16 13:30:14.696504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.696416    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerName="main"
Sep 16 13:30:14.696504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.696469    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa949965-69f8-4118-942e-df4a9d38e31a" containerName="main"
Sep 16 13:30:14.696504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.696476    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f6d3f4-9865-4675-a192-b7b3cb4a3b26" containerName="main"
Sep 16 13:30:14.698278 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.698262    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.702558 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.702535    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 13:30:14.702839 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.702819    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 13:30:14.702941 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.702837    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 13:30:14.702998 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.702937    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-qt9sm\""
Sep 16 13:30:14.711777 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.711757    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"]
Sep 16 13:30:14.819428 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.819399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.819558 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.819431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rnn\" (UniqueName: \"kubernetes.io/projected/f95b3089-2a83-49cd-932e-6778c8080859-kube-api-access-j9rnn\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.819558 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.819463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95b3089-2a83-49cd-932e-6778c8080859-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.819637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.819552    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.819637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.819594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.819637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.819620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.882437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.882398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:14.920219 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.920194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.920339 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.920230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.920339 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.920247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rnn\" (UniqueName: \"kubernetes.io/projected/f95b3089-2a83-49cd-932e-6778c8080859-kube-api-access-j9rnn\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.920339 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.920277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95b3089-2a83-49cd-932e-6778c8080859-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.920339 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.920310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.920546 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.920342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.920733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.920713    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.920807 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.920769    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.920807 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.920774    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.922523 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.922507    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.922807 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.922790    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95b3089-2a83-49cd-932e-6778c8080859-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:14.928954 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:14.928932    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rnn\" (UniqueName: \"kubernetes.io/projected/f95b3089-2a83-49cd-932e-6778c8080859-kube-api-access-j9rnn\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:15.007962 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:15.007911    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:15.285223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:15.285197    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"]
Sep 16 13:30:15.287266 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:30:15.287233    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95b3089_2a83_49cd_932e_6778c8080859.slice/crio-3b8678064ce041e148299e82eca7ef2d050f89e2f3a04ed2c34e0f44064ab61b WatchSource:0}: Error finding container 3b8678064ce041e148299e82eca7ef2d050f89e2f3a04ed2c34e0f44064ab61b: Status 404 returned error can't find the container with id 3b8678064ce041e148299e82eca7ef2d050f89e2f3a04ed2c34e0f44064ab61b
Sep 16 13:30:15.368831 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:15.368802    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh" event={"ID":"f95b3089-2a83-49cd-932e-6778c8080859","Type":"ContainerStarted","Data":"36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b"}
Sep 16 13:30:15.368942 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:15.368860    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh" event={"ID":"f95b3089-2a83-49cd-932e-6778c8080859","Type":"ContainerStarted","Data":"3b8678064ce041e148299e82eca7ef2d050f89e2f3a04ed2c34e0f44064ab61b"}
Sep 16 13:30:15.389975 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:15.389931    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh" podStartSLOduration=1.389912789 podStartE2EDuration="1.389912789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:30:15.387459738 +0000 UTC m=+2536.568602397" watchObservedRunningTime="2026-09-16 13:30:15.389912789 +0000 UTC m=+2536.571055442"
Sep 16 13:30:16.008319 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:16.008282    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:16.777710 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:16.777680    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:19.106579 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:19.106538    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:23.079529 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:23.079496    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:25.008141 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:25.008103    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:25.020581 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:25.020550    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:25.094384 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:25.094349    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:25.432551 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:25.432524    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:30:40.726904 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:40.726871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:40.827426 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:40.827384    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:40.921964 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:40.921927    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:41.412182 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:41.412126    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:45.823517 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:45.823471    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:30:51.615913 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:30:51.615871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:02.071147 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:02.071106    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:06.618055 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:06.618018    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:22.962182 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:22.962093    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:25.964873 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:25.964836    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:26.064908 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:26.064867    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:28.794683 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:28.794632    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:28.894720 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:28.894689    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:43.342035 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.342003    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:43.355468 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.355443    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"]
Sep 16 13:31:43.365292 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:31:43.365261    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:31:43.365292 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:31:43.365292 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:31:43.365292 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b" cmd=["/bin/sleep","15"]
Sep 16 13:31:43.365555 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.365307    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh" message="" err=<
Sep 16 13:31:43.365555 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:31:43.365555 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:31:43.365555 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:31:43.365555 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:31:43.365319    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:31:43.365555 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:31:43.365555 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:31:43.365555 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh" podUID="f95b3089-2a83-49cd-932e-6778c8080859" containerName="main" containerID="cri-o://36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b"
Sep 16 13:31:43.365555 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.365368    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh" podUID="f95b3089-2a83-49cd-932e-6778c8080859" containerName="main" containerID="cri-o://36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b" gracePeriod=60
Sep 16 13:31:43.402597 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.402568    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:31:43.419575 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.419547    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh" podUID="f95b3089-2a83-49cd-932e-6778c8080859" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 13:31:43.707335 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.707307    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:31:43.720757 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.720736    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-model-cache\") pod \"f95b3089-2a83-49cd-932e-6778c8080859\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") "
Sep 16 13:31:43.720843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.720772    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95b3089-2a83-49cd-932e-6778c8080859-tls-certs\") pod \"f95b3089-2a83-49cd-932e-6778c8080859\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") "
Sep 16 13:31:43.720843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.720791    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9rnn\" (UniqueName: \"kubernetes.io/projected/f95b3089-2a83-49cd-932e-6778c8080859-kube-api-access-j9rnn\") pod \"f95b3089-2a83-49cd-932e-6778c8080859\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") "
Sep 16 13:31:43.720843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.720822    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-tmp-dir\") pod \"f95b3089-2a83-49cd-932e-6778c8080859\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") "
Sep 16 13:31:43.720996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.720884    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-dshm\") pod \"f95b3089-2a83-49cd-932e-6778c8080859\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") "
Sep 16 13:31:43.720996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.720936    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-home\") pod \"f95b3089-2a83-49cd-932e-6778c8080859\" (UID: \"f95b3089-2a83-49cd-932e-6778c8080859\") "
Sep 16 13:31:43.721104 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.721015    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-model-cache" (OuterVolumeSpecName: "model-cache") pod "f95b3089-2a83-49cd-932e-6778c8080859" (UID: "f95b3089-2a83-49cd-932e-6778c8080859"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:31:43.721104 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.721086    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f95b3089-2a83-49cd-932e-6778c8080859" (UID: "f95b3089-2a83-49cd-932e-6778c8080859"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:31:43.721218 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.721189    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:31:43.721218 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.721206    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:31:43.721288 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.721213    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-home" (OuterVolumeSpecName: "home") pod "f95b3089-2a83-49cd-932e-6778c8080859" (UID: "f95b3089-2a83-49cd-932e-6778c8080859"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:31:43.723138 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.723111    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95b3089-2a83-49cd-932e-6778c8080859-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f95b3089-2a83-49cd-932e-6778c8080859" (UID: "f95b3089-2a83-49cd-932e-6778c8080859"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:31:43.723225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.723147    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-dshm" (OuterVolumeSpecName: "dshm") pod "f95b3089-2a83-49cd-932e-6778c8080859" (UID: "f95b3089-2a83-49cd-932e-6778c8080859"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:31:43.723225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.723168    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95b3089-2a83-49cd-932e-6778c8080859-kube-api-access-j9rnn" (OuterVolumeSpecName: "kube-api-access-j9rnn") pod "f95b3089-2a83-49cd-932e-6778c8080859" (UID: "f95b3089-2a83-49cd-932e-6778c8080859"). InnerVolumeSpecName "kube-api-access-j9rnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:31:43.821616 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.821580    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f95b3089-2a83-49cd-932e-6778c8080859-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:31:43.821616 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.821616    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rnn\" (UniqueName: \"kubernetes.io/projected/f95b3089-2a83-49cd-932e-6778c8080859-kube-api-access-j9rnn\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:31:43.821846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.821629    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:31:43.821846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.821669    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f95b3089-2a83-49cd-932e-6778c8080859-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:31:43.822316 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.822297    2575 generic.go:356] "Generic (PLEG): container finished" podID="f95b3089-2a83-49cd-932e-6778c8080859" containerID="36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b" exitCode=0
Sep 16 13:31:43.822382 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.822347    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"
Sep 16 13:31:43.822423 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.822383    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh" event={"ID":"f95b3089-2a83-49cd-932e-6778c8080859","Type":"ContainerDied","Data":"36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b"}
Sep 16 13:31:43.822459 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.822442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh" event={"ID":"f95b3089-2a83-49cd-932e-6778c8080859","Type":"ContainerDied","Data":"3b8678064ce041e148299e82eca7ef2d050f89e2f3a04ed2c34e0f44064ab61b"}
Sep 16 13:31:43.822499 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.822464    2575 scope.go:117] "RemoveContainer" containerID="36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b"
Sep 16 13:31:43.834204 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.834185    2575 scope.go:117] "RemoveContainer" containerID="36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b"
Sep 16 13:31:43.834471 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:31:43.834452    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b\": container with ID starting with 36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b not found: ID does not exist" containerID="36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b"
Sep 16 13:31:43.834535 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.834480    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b"} err="failed to get container status \"36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b\": rpc error: code = NotFound desc = could not find container \"36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b\": container with ID starting with 36f57368bffc2d04138e34418acd996b5c0a09ac2a90a10b19e54cb495a6c22b not found: ID does not exist"
Sep 16 13:31:43.845151 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.845129    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"]
Sep 16 13:31:43.849659 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:43.849628    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ttvmh"]
Sep 16 13:31:45.328421 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:31:45.328385    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f95b3089-2a83-49cd-932e-6778c8080859" path="/var/lib/kubelet/pods/f95b3089-2a83-49cd-932e-6778c8080859/volumes"
Sep 16 13:32:30.078948 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:30.078914    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:30.173345 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:30.173311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:30.283830 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:30.283795    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:31.176312 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:31.176247    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:31.276223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:31.276191    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:31.551567 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:31.551476    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:36.265277 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:36.265242    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:36.639029 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:36.639002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:42.097012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.096981    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"]
Sep 16 13:32:42.097440 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.097348    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f95b3089-2a83-49cd-932e-6778c8080859" containerName="main"
Sep 16 13:32:42.097440 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.097360    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f95b3089-2a83-49cd-932e-6778c8080859" containerName="main"
Sep 16 13:32:42.097440 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.097411    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f95b3089-2a83-49cd-932e-6778c8080859" containerName="main"
Sep 16 13:32:42.100435 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.100418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.103408 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.103387    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 13:32:42.104788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.104771    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 13:32:42.104858 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.104805    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-kp5mb\""
Sep 16 13:32:42.104915 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.104877    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 13:32:42.112018 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.111997    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"]
Sep 16 13:32:42.133167 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.133143    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:42.272093 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.272063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2pz\" (UniqueName: \"kubernetes.io/projected/ae5a763c-70de-49c4-a0ac-ec22c88037da-kube-api-access-bf2pz\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.272222 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.272122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.272222 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.272190    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.272311 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.272243    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5a763c-70de-49c4-a0ac-ec22c88037da-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.272311 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.272278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.272311 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.272304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.342202 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.342165    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:42.372998 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.372972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2pz\" (UniqueName: \"kubernetes.io/projected/ae5a763c-70de-49c4-a0ac-ec22c88037da-kube-api-access-bf2pz\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.373150 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.373012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.373150 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.373034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.373265 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.373218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5a763c-70de-49c4-a0ac-ec22c88037da-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.373326 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.373268    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.373402 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.373376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.373464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.373416    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.373464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.373452    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.373698 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.373669    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.375731 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.375711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.375940 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.375924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5a763c-70de-49c4-a0ac-ec22c88037da-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.381441 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.381421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2pz\" (UniqueName: \"kubernetes.io/projected/ae5a763c-70de-49c4-a0ac-ec22c88037da-kube-api-access-bf2pz\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.411453 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.411429    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:42.695267 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:42.695237    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"]
Sep 16 13:32:42.696734 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:32:42.696703    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5a763c_70de_49c4_a0ac_ec22c88037da.slice/crio-5627fb401daf4441f29d0565248a05e618832cf2de6f307440c3efa88a2d0486 WatchSource:0}: Error finding container 5627fb401daf4441f29d0565248a05e618832cf2de6f307440c3efa88a2d0486: Status 404 returned error can't find the container with id 5627fb401daf4441f29d0565248a05e618832cf2de6f307440c3efa88a2d0486
Sep 16 13:32:43.149520 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:43.149487    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp" event={"ID":"ae5a763c-70de-49c4-a0ac-ec22c88037da","Type":"ContainerStarted","Data":"def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839"}
Sep 16 13:32:43.149943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:43.149540    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp" event={"ID":"ae5a763c-70de-49c4-a0ac-ec22c88037da","Type":"ContainerStarted","Data":"5627fb401daf4441f29d0565248a05e618832cf2de6f307440c3efa88a2d0486"}
Sep 16 13:32:43.171283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:43.171235    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp" podStartSLOduration=1.171222167 podStartE2EDuration="1.171222167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:32:43.167939736 +0000 UTC m=+2684.349082402" watchObservedRunningTime="2026-09-16 13:32:43.171222167 +0000 UTC m=+2684.352364819"
Sep 16 13:32:43.200988 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:43.200956    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:43.303866 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:43.303824    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:43.411830 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:43.411762    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:44.588595 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:44.588560    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:45.382278 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:45.382240    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:51.711326 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:51.711238    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:32:52.412398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:52.412362    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:52.425390 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:52.425368    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:32:53.211801 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:32:53.211765    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:33:11.701704 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:33:11.701665    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:14.109466 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:33:14.109426    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:19.781152 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:33:19.781114    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:19.852921 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:33:19.852889    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:20.703360 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:33:20.703326    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:04.927118 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:04.927082    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"]
Sep 16 13:34:04.930702 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:04.930682    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:04.933309 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:04.933289    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 13:34:04.941297 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:04.941274    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"]
Sep 16 13:34:04.994596 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:04.994564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:05.027751 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.027721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.027894 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.027756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.027894 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.027779    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.027894 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.027858    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.027894 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.027882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9krf\" (UniqueName: \"kubernetes.io/projected/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-kube-api-access-q9krf\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.028035 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.027913    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.128751 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.128724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.128903 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.128763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9krf\" (UniqueName: \"kubernetes.io/projected/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-kube-api-access-q9krf\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.128903 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.128804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.128903 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.128832    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.128903 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.128851    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.128903 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.128878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.129240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.129220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.129340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.129234    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.129340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.129283    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.131132 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.131111    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.131444 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.131428    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.138732 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.138703    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9krf\" (UniqueName: \"kubernetes.io/projected/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-kube-api-access-q9krf\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.195544 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.195465    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:05.241701 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.241671    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:05.519315 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.519287    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"]
Sep 16 13:34:05.521172 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:34:05.521146    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110a8ff2_aed4_47ee_ba3d_4de4fe4c213c.slice/crio-91440beb9bf4f6666711c772cd9e8937bfe50129ccc4acdfbca22839801da4f3 WatchSource:0}: Error finding container 91440beb9bf4f6666711c772cd9e8937bfe50129ccc4acdfbca22839801da4f3: Status 404 returned error can't find the container with id 91440beb9bf4f6666711c772cd9e8937bfe50129ccc4acdfbca22839801da4f3
Sep 16 13:34:05.574059 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:05.574031    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz" event={"ID":"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c","Type":"ContainerStarted","Data":"91440beb9bf4f6666711c772cd9e8937bfe50129ccc4acdfbca22839801da4f3"}
Sep 16 13:34:06.580765 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:06.580723    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz" event={"ID":"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c","Type":"ContainerStarted","Data":"de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075"}
Sep 16 13:34:06.605317 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:06.605263    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz" podStartSLOduration=2.605248311 podStartE2EDuration="2.605248311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:06.603749144 +0000 UTC m=+2767.784891797" watchObservedRunningTime="2026-09-16 13:34:06.605248311 +0000 UTC m=+2767.786390964"
Sep 16 13:34:07.241841 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:07.241809    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:14.919534 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:14.919491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:15.242970 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:15.242888    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:15.255389 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:15.255361    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:15.637361 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:15.637326    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:22.737288 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:22.737208    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:40.206449 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.206419    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"]
Sep 16 13:34:40.215456 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.215414    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:40.216468 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:34:40.216438    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:34:40.216468 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:34:40.216468 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:34:40.216468 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075" cmd=["/bin/sleep","15"]
Sep 16 13:34:40.216763 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.216491    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz" message="" err=<
Sep 16 13:34:40.216763 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:34:40.216763 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:34:40.216763 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:34:40.216763 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:34:40.216507    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:34:40.216763 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:34:40.216763 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:34:40.216763 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz" podUID="110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" containerName="main" containerID="cri-o://de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075"
Sep 16 13:34:40.216763 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.216552    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz" podUID="110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" containerName="main" containerID="cri-o://de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075" gracePeriod=60
Sep 16 13:34:40.282387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.282356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:40.464891 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.464832    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:40.584564 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.584535    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-dshm\") pod \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") "
Sep 16 13:34:40.584772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.584576    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9krf\" (UniqueName: \"kubernetes.io/projected/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-kube-api-access-q9krf\") pod \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") "
Sep 16 13:34:40.584772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.584621    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-model-cache\") pod \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") "
Sep 16 13:34:40.584772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.584683    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tmp-dir\") pod \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") "
Sep 16 13:34:40.584772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.584769    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-home\") pod \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") "
Sep 16 13:34:40.585012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.584840    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tls-certs\") pod \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\" (UID: \"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c\") "
Sep 16 13:34:40.585012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.584934    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-model-cache" (OuterVolumeSpecName: "model-cache") pod "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" (UID: "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:34:40.585012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.584946    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" (UID: "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:34:40.585169 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.585054    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-home" (OuterVolumeSpecName: "home") pod "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" (UID: "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:34:40.585169 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.585094    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:34:40.585169 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.585112    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:34:40.587090 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.587051    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" (UID: "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:34:40.587090 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.587085    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-dshm" (OuterVolumeSpecName: "dshm") pod "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" (UID: "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:34:40.587242 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.587129    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-kube-api-access-q9krf" (OuterVolumeSpecName: "kube-api-access-q9krf") pod "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" (UID: "110a8ff2-aed4-47ee-ba3d-4de4fe4c213c"). InnerVolumeSpecName "kube-api-access-q9krf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:34:40.686445 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.686418    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:34:40.686445 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.686443    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:34:40.686622 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.686452    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9krf\" (UniqueName: \"kubernetes.io/projected/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-kube-api-access-q9krf\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:34:40.686622 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.686462    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:34:40.760289 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.760215    2575 generic.go:356] "Generic (PLEG): container finished" podID="110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" containerID="de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075" exitCode=0
Sep 16 13:34:40.760289 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.760249    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz" event={"ID":"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c","Type":"ContainerDied","Data":"de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075"}
Sep 16 13:34:40.760289 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.760268    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"
Sep 16 13:34:40.760289 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.760285    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz" event={"ID":"110a8ff2-aed4-47ee-ba3d-4de4fe4c213c","Type":"ContainerDied","Data":"91440beb9bf4f6666711c772cd9e8937bfe50129ccc4acdfbca22839801da4f3"}
Sep 16 13:34:40.760608 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.760306    2575 scope.go:117] "RemoveContainer" containerID="de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075"
Sep 16 13:34:40.770030 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.770008    2575 scope.go:117] "RemoveContainer" containerID="de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075"
Sep 16 13:34:40.770338 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:34:40.770318    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075\": container with ID starting with de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075 not found: ID does not exist" containerID="de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075"
Sep 16 13:34:40.770406 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.770346    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075"} err="failed to get container status \"de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075\": rpc error: code = NotFound desc = could not find container \"de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075\": container with ID starting with de072db2624878e4ec6c0bfdcf7c12222bd009902206b03257e5ac0aad2a6075 not found: ID does not exist"
Sep 16 13:34:40.782636 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.782606    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"]
Sep 16 13:34:40.788361 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:40.788336    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995274kz"]
Sep 16 13:34:41.329015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:41.328979    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" path="/var/lib/kubelet/pods/110a8ff2-aed4-47ee-ba3d-4de4fe4c213c/volumes"
Sep 16 13:34:46.405583 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.405546    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:46.762489 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.762462    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"]
Sep 16 13:34:46.762867 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.762854    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" containerName="main"
Sep 16 13:34:46.762923 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.762870    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" containerName="main"
Sep 16 13:34:46.762960 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.762924    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="110a8ff2-aed4-47ee-ba3d-4de4fe4c213c" containerName="main"
Sep 16 13:34:46.766938 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.766918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.771770 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.771623    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 16 13:34:46.771888 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.771850    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 13:34:46.779000 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.778981    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-5prt2\""
Sep 16 13:34:46.779113 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.779063    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 13:34:46.805903 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.805877    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"]
Sep 16 13:34:46.839599 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.839567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj767\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-kube-api-access-zj767\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.839758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.839607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.839758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.839636    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.839758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.839721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.839758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.839750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.839898 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.839773    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.839898 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.839798    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476795b-e966-4cfe-a8af-ba1fc4610bda-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.839967 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.839889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.839967 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.839914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941238 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941238 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941476 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj767\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-kube-api-access-zj767\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941476 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941476 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941476 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941476 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941476 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941476 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476795b-e966-4cfe-a8af-ba1fc4610bda-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941878 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941817    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941932 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941870    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.941932 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941910    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.942046 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.941935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.942268 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.942097    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476795b-e966-4cfe-a8af-ba1fc4610bda-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.943810 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.943790    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.944032 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.944014    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.950384 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.950363    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:46.951528 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:46.951512    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj767\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-kube-api-access-zj767\") pod \"router-gateway-1-openshift-default-55f8b478f9-qcnqd\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:47.062203 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:47.062118    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:47.077014 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:47.076986    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:47.187532 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:47.187498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:47.372425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:47.372402    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"]
Sep 16 13:34:47.374255 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:34:47.374232    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0476795b_e966_4cfe_a8af_ba1fc4610bda.slice/crio-1de34b56c516d0a974736fe85e15ac6288abfd1b43dc017a5bccb2227806d751 WatchSource:0}: Error finding container 1de34b56c516d0a974736fe85e15ac6288abfd1b43dc017a5bccb2227806d751: Status 404 returned error can't find the container with id 1de34b56c516d0a974736fe85e15ac6288abfd1b43dc017a5bccb2227806d751
Sep 16 13:34:47.376168 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:47.376151    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:34:47.798768 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:47.798683    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" event={"ID":"0476795b-e966-4cfe-a8af-ba1fc4610bda","Type":"ContainerStarted","Data":"1de34b56c516d0a974736fe85e15ac6288abfd1b43dc017a5bccb2227806d751"}
Sep 16 13:34:50.033150 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:50.033046    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:50.376422 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:50.376385    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:34:50.376534 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:50.376461    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:34:50.376534 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:50.376493    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:34:50.825253 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:50.825154    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" event={"ID":"0476795b-e966-4cfe-a8af-ba1fc4610bda","Type":"ContainerStarted","Data":"9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a"}
Sep 16 13:34:50.856871 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:50.856814    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" podStartSLOduration=1.8568925109999999 podStartE2EDuration="4.856795555s" totalImagesPullingTime="2.999903044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:50.852290217 +0000 UTC m=+2812.033432871" watchObservedRunningTime="2026-09-16 13:34:50.856795555 +0000 UTC m=+2812.037938209"
Sep 16 13:34:51.077963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:51.077883    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:51.082479 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:51.082452    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:51.435943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:51.435891    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:51.830966 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:51.830882    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:51.831701 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:51.831680    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:34:57.584281 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:34:57.584250    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:04.290758 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.290724    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"]
Sep 16 13:35:04.296469 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.296438    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.305630 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.305602    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 16 13:35:04.305884 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.305618    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-kk588\""
Sep 16 13:35:04.308162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.308142    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"]
Sep 16 13:35:04.340669 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.340610    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:04.393877 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.393842    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.394020 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.393891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nm2\" (UniqueName: \"kubernetes.io/projected/2795213f-9bcb-4cdf-80b4-17faceb9856c-kube-api-access-r7nm2\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.394020 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.393924    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.394020 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.393956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.394020 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.393981    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.394161 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.394024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2795213f-9bcb-4cdf-80b4-17faceb9856c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.495266 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.495236    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2795213f-9bcb-4cdf-80b4-17faceb9856c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.495418 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.495288    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.495418 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.495330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nm2\" (UniqueName: \"kubernetes.io/projected/2795213f-9bcb-4cdf-80b4-17faceb9856c-kube-api-access-r7nm2\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.495418 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.495356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.495418 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.495389    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.495601 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.495532    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.495795 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.495770    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.495905 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.495815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.495947 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.495931    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.497799 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.497778    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.498055 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.498040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2795213f-9bcb-4cdf-80b4-17faceb9856c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.503900 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.503881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nm2\" (UniqueName: \"kubernetes.io/projected/2795213f-9bcb-4cdf-80b4-17faceb9856c-kube-api-access-r7nm2\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.608337 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.608268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:04.886537 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.886511    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"]
Sep 16 13:35:04.888500 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:35:04.888474    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2795213f_9bcb_4cdf_80b4_17faceb9856c.slice/crio-ed7b608e16ca6ad5655192f84352636814307f83791363a0a883c2efdeae9b79 WatchSource:0}: Error finding container ed7b608e16ca6ad5655192f84352636814307f83791363a0a883c2efdeae9b79: Status 404 returned error can't find the container with id ed7b608e16ca6ad5655192f84352636814307f83791363a0a883c2efdeae9b79
Sep 16 13:35:04.898068 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:04.898037    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff" event={"ID":"2795213f-9bcb-4cdf-80b4-17faceb9856c","Type":"ContainerStarted","Data":"ed7b608e16ca6ad5655192f84352636814307f83791363a0a883c2efdeae9b79"}
Sep 16 13:35:05.905382 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:05.905341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff" event={"ID":"2795213f-9bcb-4cdf-80b4-17faceb9856c","Type":"ContainerStarted","Data":"9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894"}
Sep 16 13:35:06.609105 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:06.609069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:14.609353 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.609275    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:14.621452 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.621431    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:14.644812 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.644768    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff" podStartSLOduration=10.644754085 podStartE2EDuration="10.644754085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:35:05.934693442 +0000 UTC m=+2827.115836096" watchObservedRunningTime="2026-09-16 13:35:14.644754085 +0000 UTC m=+2835.825897104"
Sep 16 13:35:14.958469 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.958444    2575 generic.go:356] "Generic (PLEG): container finished" podID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerID="9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a" exitCode=0
Sep 16 13:35:14.958603 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.958514    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" event={"ID":"0476795b-e966-4cfe-a8af-ba1fc4610bda","Type":"ContainerDied","Data":"9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a"}
Sep 16 13:35:14.959010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.958993    2575 scope.go:117] "RemoveContainer" containerID="9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a"
Sep 16 13:35:14.959936 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.959908    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:35:14.960005 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.959975    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:35:14.960045 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.960004    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:35:14.970307 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:14.970290    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:35:15.966981 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:15.966940    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" event={"ID":"0476795b-e966-4cfe-a8af-ba1fc4610bda","Type":"ContainerStarted","Data":"4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b"}
Sep 16 13:35:16.077197 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:16.077161    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:35:16.077387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:16.077362    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 13:35:17.077870 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:17.077828    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 13:35:18.077987 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:18.077941    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 13:35:19.077787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:19.077742    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 13:35:20.077486 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:20.077444    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 16 13:35:21.081278 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:21.081250    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:35:21.081684 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:21.081542    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:35:21.082342 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:21.082321    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:35:28.880491 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:28.880450    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:30.625565 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:30.625412    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:30.643099 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:30.643071    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"]
Sep 16 13:35:30.651845 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:35:30.651816    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:35:30.651845 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:35:30.651845 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:35:30.651845 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839" cmd=["/bin/sleep","15"]
Sep 16 13:35:30.652069 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:30.651860    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp" message="" err=<
Sep 16 13:35:30.652069 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:35:30.652069 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:35:30.652069 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:35:30.652069 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:35:30.651874    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:35:30.652069 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:35:30.652069 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:35:30.652069 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp" podUID="ae5a763c-70de-49c4-a0ac-ec22c88037da" containerName="main" containerID="cri-o://def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839"
Sep 16 13:35:30.652069 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:30.651928    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp" podUID="ae5a763c-70de-49c4-a0ac-ec22c88037da" containerName="main" containerID="cri-o://def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839" gracePeriod=60
Sep 16 13:35:30.719909 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:30.719874    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:31.007950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.007928    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:35:31.036428 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036399    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-home\") pod \"ae5a763c-70de-49c4-a0ac-ec22c88037da\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") "
Sep 16 13:35:31.036428 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036428    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-tmp-dir\") pod \"ae5a763c-70de-49c4-a0ac-ec22c88037da\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") "
Sep 16 13:35:31.036591 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036450    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-dshm\") pod \"ae5a763c-70de-49c4-a0ac-ec22c88037da\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") "
Sep 16 13:35:31.036591 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036490    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf2pz\" (UniqueName: \"kubernetes.io/projected/ae5a763c-70de-49c4-a0ac-ec22c88037da-kube-api-access-bf2pz\") pod \"ae5a763c-70de-49c4-a0ac-ec22c88037da\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") "
Sep 16 13:35:31.036591 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036540    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5a763c-70de-49c4-a0ac-ec22c88037da-tls-certs\") pod \"ae5a763c-70de-49c4-a0ac-ec22c88037da\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") "
Sep 16 13:35:31.036591 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036578    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-model-cache\") pod \"ae5a763c-70de-49c4-a0ac-ec22c88037da\" (UID: \"ae5a763c-70de-49c4-a0ac-ec22c88037da\") "
Sep 16 13:35:31.036845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036665    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-home" (OuterVolumeSpecName: "home") pod "ae5a763c-70de-49c4-a0ac-ec22c88037da" (UID: "ae5a763c-70de-49c4-a0ac-ec22c88037da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:35:31.036845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036765    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae5a763c-70de-49c4-a0ac-ec22c88037da" (UID: "ae5a763c-70de-49c4-a0ac-ec22c88037da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:35:31.036958 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036927    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:35:31.036958 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036946    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:35:31.037055 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.036972    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae5a763c-70de-49c4-a0ac-ec22c88037da" (UID: "ae5a763c-70de-49c4-a0ac-ec22c88037da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:35:31.038876 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.038850    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5a763c-70de-49c4-a0ac-ec22c88037da-kube-api-access-bf2pz" (OuterVolumeSpecName: "kube-api-access-bf2pz") pod "ae5a763c-70de-49c4-a0ac-ec22c88037da" (UID: "ae5a763c-70de-49c4-a0ac-ec22c88037da"). InnerVolumeSpecName "kube-api-access-bf2pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:35:31.039244 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.039218    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-dshm" (OuterVolumeSpecName: "dshm") pod "ae5a763c-70de-49c4-a0ac-ec22c88037da" (UID: "ae5a763c-70de-49c4-a0ac-ec22c88037da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:35:31.039355 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.039270    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae5a763c-70de-49c4-a0ac-ec22c88037da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae5a763c-70de-49c4-a0ac-ec22c88037da" (UID: "ae5a763c-70de-49c4-a0ac-ec22c88037da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:35:31.047705 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.047677    2575 generic.go:356] "Generic (PLEG): container finished" podID="ae5a763c-70de-49c4-a0ac-ec22c88037da" containerID="def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839" exitCode=0
Sep 16 13:35:31.047864 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.047734    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"
Sep 16 13:35:31.047864 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.047804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp" event={"ID":"ae5a763c-70de-49c4-a0ac-ec22c88037da","Type":"ContainerDied","Data":"def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839"}
Sep 16 13:35:31.047864 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.047848    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp" event={"ID":"ae5a763c-70de-49c4-a0ac-ec22c88037da","Type":"ContainerDied","Data":"5627fb401daf4441f29d0565248a05e618832cf2de6f307440c3efa88a2d0486"}
Sep 16 13:35:31.048052 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.047874    2575 scope.go:117] "RemoveContainer" containerID="def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839"
Sep 16 13:35:31.060254 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.060238    2575 scope.go:117] "RemoveContainer" containerID="def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839"
Sep 16 13:35:31.060586 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:35:31.060567    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839\": container with ID starting with def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839 not found: ID does not exist" containerID="def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839"
Sep 16 13:35:31.060760 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.060594    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839"} err="failed to get container status \"def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839\": rpc error: code = NotFound desc = could not find container \"def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839\": container with ID starting with def4f7f80ed4bca0951cab5c615d690ad8bb8c31adb91ef88ee4e325d38a4839 not found: ID does not exist"
Sep 16 13:35:31.072904 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.072880    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"]
Sep 16 13:35:31.076894 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.076873    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-7hdpp"]
Sep 16 13:35:31.138342 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.138312    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae5a763c-70de-49c4-a0ac-ec22c88037da-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:35:31.138342 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.138340    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:35:31.138468 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.138352    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae5a763c-70de-49c4-a0ac-ec22c88037da-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:35:31.138468 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.138367    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf2pz\" (UniqueName: \"kubernetes.io/projected/ae5a763c-70de-49c4-a0ac-ec22c88037da-kube-api-access-bf2pz\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:35:31.330016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:31.329938    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae5a763c-70de-49c4-a0ac-ec22c88037da" path="/var/lib/kubelet/pods/ae5a763c-70de-49c4-a0ac-ec22c88037da/volumes"
Sep 16 13:35:35.057841 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:35.057803    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:35.195010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:35.194979    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:54.226547 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:35:54.226512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:20.562847 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:36:20.562787    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:20.695321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:36:20.695286    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:27.374725 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:36:27.374686    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:33.632404 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:36:33.632352    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:33.761583 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:36:33.761553    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:46.947921 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:36:46.947877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:47.960986 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:36:47.960945    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:48.216466 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:36:48.216355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:59.752195 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:36:59.752158    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:30.797136 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.797057    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"]
Sep 16 13:37:30.797666 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.797430    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae5a763c-70de-49c4-a0ac-ec22c88037da" containerName="main"
Sep 16 13:37:30.797666 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.797441    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae5a763c-70de-49c4-a0ac-ec22c88037da" containerName="main"
Sep 16 13:37:30.797666 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.797522    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae5a763c-70de-49c4-a0ac-ec22c88037da" containerName="main"
Sep 16 13:37:30.800943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.800922    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:30.805198 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.805171    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 13:37:30.806532 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.806512    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 13:37:30.806676 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.806557    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 13:37:30.806770 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.806690    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-ncqxw\""
Sep 16 13:37:30.833901 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.833869    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"]
Sep 16 13:37:30.847154 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.847122    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:30.968547 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.968518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:30.968547 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.968549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81acc17-e750-4b05-88e8-f7418db89084-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:30.968817 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.968573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:30.968817 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.968712    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:30.968817 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.968756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:30.968953 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:30.968839    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ld2j\" (UniqueName: \"kubernetes.io/projected/f81acc17-e750-4b05-88e8-f7418db89084-kube-api-access-6ld2j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.069996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.069917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.069996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.069958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.070208 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.070007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ld2j\" (UniqueName: \"kubernetes.io/projected/f81acc17-e750-4b05-88e8-f7418db89084-kube-api-access-6ld2j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.070208 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.070037    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.070208 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.070062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81acc17-e750-4b05-88e8-f7418db89084-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.070675 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.070214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.070675 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.070434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.070675 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.070610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.070863 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.070695    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.072510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.072484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.072795 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.072778    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81acc17-e750-4b05-88e8-f7418db89084-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.078907 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.078887    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ld2j\" (UniqueName: \"kubernetes.io/projected/f81acc17-e750-4b05-88e8-f7418db89084-kube-api-access-6ld2j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.089997 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.089974    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:31.111083 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.111061    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:31.398308 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.398281    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"]
Sep 16 13:37:31.399040 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:37:31.399015    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81acc17_e750_4b05_88e8_f7418db89084.slice/crio-63a77a9b6a03ee52c7a37c97493df18d4e9591e917992cffb861f982dc11c7ae WatchSource:0}: Error finding container 63a77a9b6a03ee52c7a37c97493df18d4e9591e917992cffb861f982dc11c7ae: Status 404 returned error can't find the container with id 63a77a9b6a03ee52c7a37c97493df18d4e9591e917992cffb861f982dc11c7ae
Sep 16 13:37:31.666846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.666761    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc" event={"ID":"f81acc17-e750-4b05-88e8-f7418db89084","Type":"ContainerStarted","Data":"2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561"}
Sep 16 13:37:31.666846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.666814    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc" event={"ID":"f81acc17-e750-4b05-88e8-f7418db89084","Type":"ContainerStarted","Data":"63a77a9b6a03ee52c7a37c97493df18d4e9591e917992cffb861f982dc11c7ae"}
Sep 16 13:37:31.686968 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:31.686916    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc" podStartSLOduration=1.686869696 podStartE2EDuration="1.686869696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:31.685754792 +0000 UTC m=+2972.866897444" watchObservedRunningTime="2026-09-16 13:37:31.686869696 +0000 UTC m=+2972.868012393"
Sep 16 13:37:32.112103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:32.112080    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:38.133145 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:38.133108    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:38.237539 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:38.237509    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:41.111572 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:41.111540    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:41.123929 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:41.123903    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:37:41.464253 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:41.464166    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:41.732032 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:37:41.731953    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:38:05.885759 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:05.885724    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:05.886297 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:05.886032    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"]
Sep 16 13:38:05.896226 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:38:05.896176    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:38:05.896226 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:38:05.896226 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:38:05.896226 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561" cmd=["/bin/sleep","15"]
Sep 16 13:38:05.896574 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:05.896240    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc" message="" err=<
Sep 16 13:38:05.896574 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:38:05.896574 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:38:05.896574 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:38:05.896574 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:38:05.896253    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:38:05.896574 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:38:05.896574 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:38:05.896574 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc" podUID="f81acc17-e750-4b05-88e8-f7418db89084" containerName="main" containerID="cri-o://2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561"
Sep 16 13:38:05.896574 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:05.896301    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc" podUID="f81acc17-e750-4b05-88e8-f7418db89084" containerName="main" containerID="cri-o://2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561" gracePeriod=60
Sep 16 13:38:05.954571 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:05.954531    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:06.246283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.246261    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:38:06.369837 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.369808    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-dshm\") pod \"f81acc17-e750-4b05-88e8-f7418db89084\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") "
Sep 16 13:38:06.369996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.369845    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ld2j\" (UniqueName: \"kubernetes.io/projected/f81acc17-e750-4b05-88e8-f7418db89084-kube-api-access-6ld2j\") pod \"f81acc17-e750-4b05-88e8-f7418db89084\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") "
Sep 16 13:38:06.369996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.369879    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-tmp-dir\") pod \"f81acc17-e750-4b05-88e8-f7418db89084\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") "
Sep 16 13:38:06.369996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.369933    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-home\") pod \"f81acc17-e750-4b05-88e8-f7418db89084\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") "
Sep 16 13:38:06.369996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.369982    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81acc17-e750-4b05-88e8-f7418db89084-tls-certs\") pod \"f81acc17-e750-4b05-88e8-f7418db89084\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") "
Sep 16 13:38:06.370218 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.370039    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-model-cache\") pod \"f81acc17-e750-4b05-88e8-f7418db89084\" (UID: \"f81acc17-e750-4b05-88e8-f7418db89084\") "
Sep 16 13:38:06.370218 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.370162    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f81acc17-e750-4b05-88e8-f7418db89084" (UID: "f81acc17-e750-4b05-88e8-f7418db89084"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:06.370334 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.370233    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-home" (OuterVolumeSpecName: "home") pod "f81acc17-e750-4b05-88e8-f7418db89084" (UID: "f81acc17-e750-4b05-88e8-f7418db89084"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:06.370334 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.370315    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:38:06.370334 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.370331    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:38:06.370437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.370328    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-model-cache" (OuterVolumeSpecName: "model-cache") pod "f81acc17-e750-4b05-88e8-f7418db89084" (UID: "f81acc17-e750-4b05-88e8-f7418db89084"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:06.372185 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.372157    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-dshm" (OuterVolumeSpecName: "dshm") pod "f81acc17-e750-4b05-88e8-f7418db89084" (UID: "f81acc17-e750-4b05-88e8-f7418db89084"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:06.372185 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.372161    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81acc17-e750-4b05-88e8-f7418db89084-kube-api-access-6ld2j" (OuterVolumeSpecName: "kube-api-access-6ld2j") pod "f81acc17-e750-4b05-88e8-f7418db89084" (UID: "f81acc17-e750-4b05-88e8-f7418db89084"). InnerVolumeSpecName "kube-api-access-6ld2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:38:06.372333 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.372226    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81acc17-e750-4b05-88e8-f7418db89084-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f81acc17-e750-4b05-88e8-f7418db89084" (UID: "f81acc17-e750-4b05-88e8-f7418db89084"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:38:06.470917 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.470856    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f81acc17-e750-4b05-88e8-f7418db89084-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:38:06.470917 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.470878    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:38:06.470917 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.470888    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f81acc17-e750-4b05-88e8-f7418db89084-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:38:06.470917 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.470897    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ld2j\" (UniqueName: \"kubernetes.io/projected/f81acc17-e750-4b05-88e8-f7418db89084-kube-api-access-6ld2j\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:38:06.854409 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.854328    2575 generic.go:356] "Generic (PLEG): container finished" podID="f81acc17-e750-4b05-88e8-f7418db89084" containerID="2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561" exitCode=0
Sep 16 13:38:06.854409 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.854380    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"
Sep 16 13:38:06.854589 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.854421    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc" event={"ID":"f81acc17-e750-4b05-88e8-f7418db89084","Type":"ContainerDied","Data":"2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561"}
Sep 16 13:38:06.854589 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.854476    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc" event={"ID":"f81acc17-e750-4b05-88e8-f7418db89084","Type":"ContainerDied","Data":"63a77a9b6a03ee52c7a37c97493df18d4e9591e917992cffb861f982dc11c7ae"}
Sep 16 13:38:06.854589 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.854506    2575 scope.go:117] "RemoveContainer" containerID="2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561"
Sep 16 13:38:06.870286 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.870267    2575 scope.go:117] "RemoveContainer" containerID="2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561"
Sep 16 13:38:06.870599 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:38:06.870578    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561\": container with ID starting with 2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561 not found: ID does not exist" containerID="2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561"
Sep 16 13:38:06.870698 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.870608    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561"} err="failed to get container status \"2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561\": rpc error: code = NotFound desc = could not find container \"2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561\": container with ID starting with 2938b8c2c78362ef2baf1cde8b569c4fef644fc12c3931e4c6bdae3e67d39561 not found: ID does not exist"
Sep 16 13:38:06.883149 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.883125    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"]
Sep 16 13:38:06.891003 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:06.890981    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-8q8nc"]
Sep 16 13:38:07.329523 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:07.329481    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f81acc17-e750-4b05-88e8-f7418db89084" path="/var/lib/kubelet/pods/f81acc17-e750-4b05-88e8-f7418db89084/volumes"
Sep 16 13:38:14.859836 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:14.859792    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:56.132325 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:56.132233    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:56.198119 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:38:56.198088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:26.562400 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.562365    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"]
Sep 16 13:39:26.571446 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:39:26.571407    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 13:39:26.571446 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:39:26.571446 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:39:26.571446 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894" cmd=["/bin/sleep","15"]
Sep 16 13:39:26.571729 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.571457    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff" message="" err=<
Sep 16 13:39:26.571729 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:39:26.571729 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:39:26.571729 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 13:39:26.571729 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:39:26.571469    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 13:39:26.571729 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 13:39:26.571729 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 13:39:26.571729 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff" podUID="2795213f-9bcb-4cdf-80b4-17faceb9856c" containerName="main" containerID="cri-o://9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894"
Sep 16 13:39:26.571729 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.571512    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff" podUID="2795213f-9bcb-4cdf-80b4-17faceb9856c" containerName="main" containerID="cri-o://9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894" gracePeriod=60
Sep 16 13:39:26.573773 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.573744    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:26.927590 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.927556    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:39:26.994588 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.994558    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-home\") pod \"2795213f-9bcb-4cdf-80b4-17faceb9856c\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") "
Sep 16 13:39:26.994787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.994602    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-dshm\") pod \"2795213f-9bcb-4cdf-80b4-17faceb9856c\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") "
Sep 16 13:39:26.994787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.994684    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2795213f-9bcb-4cdf-80b4-17faceb9856c-tls-certs\") pod \"2795213f-9bcb-4cdf-80b4-17faceb9856c\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") "
Sep 16 13:39:26.994787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.994719    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-model-cache\") pod \"2795213f-9bcb-4cdf-80b4-17faceb9856c\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") "
Sep 16 13:39:26.994787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.994745    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7nm2\" (UniqueName: \"kubernetes.io/projected/2795213f-9bcb-4cdf-80b4-17faceb9856c-kube-api-access-r7nm2\") pod \"2795213f-9bcb-4cdf-80b4-17faceb9856c\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") "
Sep 16 13:39:26.994787 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.994760    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-tmp-dir\") pod \"2795213f-9bcb-4cdf-80b4-17faceb9856c\" (UID: \"2795213f-9bcb-4cdf-80b4-17faceb9856c\") "
Sep 16 13:39:26.995069 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.994908    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-home" (OuterVolumeSpecName: "home") pod "2795213f-9bcb-4cdf-80b4-17faceb9856c" (UID: "2795213f-9bcb-4cdf-80b4-17faceb9856c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:26.995069 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.995001    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2795213f-9bcb-4cdf-80b4-17faceb9856c" (UID: "2795213f-9bcb-4cdf-80b4-17faceb9856c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:26.995156 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.995078    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2795213f-9bcb-4cdf-80b4-17faceb9856c" (UID: "2795213f-9bcb-4cdf-80b4-17faceb9856c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:26.996956 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.996937    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-dshm" (OuterVolumeSpecName: "dshm") pod "2795213f-9bcb-4cdf-80b4-17faceb9856c" (UID: "2795213f-9bcb-4cdf-80b4-17faceb9856c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:26.997064 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.997046    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2795213f-9bcb-4cdf-80b4-17faceb9856c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2795213f-9bcb-4cdf-80b4-17faceb9856c" (UID: "2795213f-9bcb-4cdf-80b4-17faceb9856c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:39:26.997115 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:26.997081    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2795213f-9bcb-4cdf-80b4-17faceb9856c-kube-api-access-r7nm2" (OuterVolumeSpecName: "kube-api-access-r7nm2") pod "2795213f-9bcb-4cdf-80b4-17faceb9856c" (UID: "2795213f-9bcb-4cdf-80b4-17faceb9856c"). InnerVolumeSpecName "kube-api-access-r7nm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:39:27.096016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.095942    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2795213f-9bcb-4cdf-80b4-17faceb9856c-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:27.096016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.095970    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:27.096016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.095979    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7nm2\" (UniqueName: \"kubernetes.io/projected/2795213f-9bcb-4cdf-80b4-17faceb9856c-kube-api-access-r7nm2\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:27.096016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.095987    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:27.096016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.095995    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:27.096016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.096003    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2795213f-9bcb-4cdf-80b4-17faceb9856c-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:27.286889 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.286857    2575 generic.go:356] "Generic (PLEG): container finished" podID="2795213f-9bcb-4cdf-80b4-17faceb9856c" containerID="9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894" exitCode=0
Sep 16 13:39:27.287125 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.286906    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"
Sep 16 13:39:27.287125 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.286917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff" event={"ID":"2795213f-9bcb-4cdf-80b4-17faceb9856c","Type":"ContainerDied","Data":"9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894"}
Sep 16 13:39:27.287125 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.286957    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff" event={"ID":"2795213f-9bcb-4cdf-80b4-17faceb9856c","Type":"ContainerDied","Data":"ed7b608e16ca6ad5655192f84352636814307f83791363a0a883c2efdeae9b79"}
Sep 16 13:39:27.287125 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.286979    2575 scope.go:117] "RemoveContainer" containerID="9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894"
Sep 16 13:39:27.301281 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.301258    2575 scope.go:117] "RemoveContainer" containerID="9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894"
Sep 16 13:39:27.301561 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:39:27.301541    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894\": container with ID starting with 9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894 not found: ID does not exist" containerID="9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894"
Sep 16 13:39:27.301632 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.301568    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894"} err="failed to get container status \"9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894\": rpc error: code = NotFound desc = could not find container \"9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894\": container with ID starting with 9553abb1a4fdc9d87a7eb1126e6e621aca88a079b6356c4934fa914208fe9894 not found: ID does not exist"
Sep 16 13:39:27.314044 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.314010    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"]
Sep 16 13:39:27.322329 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.322284    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997pv9ff"]
Sep 16 13:39:27.328889 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:27.328865    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2795213f-9bcb-4cdf-80b4-17faceb9856c" path="/var/lib/kubelet/pods/2795213f-9bcb-4cdf-80b4-17faceb9856c/volumes"
Sep 16 13:39:31.548866 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:31.548834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:31.647676 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:31.647608    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:36.515728 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:36.515692    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"]
Sep 16 13:39:36.516414 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:36.515911    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy" containerID="cri-o://4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b" gracePeriod=30
Sep 16 13:39:36.610447 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:36.610416    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:41.753388 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.753364    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:39:41.917786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.917755    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476795b-e966-4cfe-a8af-ba1fc4610bda-istiod-ca-cert\") pod \"0476795b-e966-4cfe-a8af-ba1fc4610bda\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") "
Sep 16 13:39:41.917786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.917791    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-podinfo\") pod \"0476795b-e966-4cfe-a8af-ba1fc4610bda\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") "
Sep 16 13:39:41.918012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.917824    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-envoy\") pod \"0476795b-e966-4cfe-a8af-ba1fc4610bda\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") "
Sep 16 13:39:41.918012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.917997    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-credential-socket\") pod \"0476795b-e966-4cfe-a8af-ba1fc4610bda\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") "
Sep 16 13:39:41.918100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918041    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-data\") pod \"0476795b-e966-4cfe-a8af-ba1fc4610bda\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") "
Sep 16 13:39:41.918100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918066    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-token\") pod \"0476795b-e966-4cfe-a8af-ba1fc4610bda\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") "
Sep 16 13:39:41.918100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918095    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-socket\") pod \"0476795b-e966-4cfe-a8af-ba1fc4610bda\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") "
Sep 16 13:39:41.918262 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918097    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0476795b-e966-4cfe-a8af-ba1fc4610bda-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0476795b-e966-4cfe-a8af-ba1fc4610bda" (UID: "0476795b-e966-4cfe-a8af-ba1fc4610bda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:39:41.918262 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918132    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-certs\") pod \"0476795b-e966-4cfe-a8af-ba1fc4610bda\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") "
Sep 16 13:39:41.918262 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918224    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj767\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-kube-api-access-zj767\") pod \"0476795b-e966-4cfe-a8af-ba1fc4610bda\" (UID: \"0476795b-e966-4cfe-a8af-ba1fc4610bda\") "
Sep 16 13:39:41.918425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918263    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0476795b-e966-4cfe-a8af-ba1fc4610bda" (UID: "0476795b-e966-4cfe-a8af-ba1fc4610bda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:41.918425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918284    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0476795b-e966-4cfe-a8af-ba1fc4610bda" (UID: "0476795b-e966-4cfe-a8af-ba1fc4610bda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:41.918425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918297    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-data" (OuterVolumeSpecName: "istio-data") pod "0476795b-e966-4cfe-a8af-ba1fc4610bda" (UID: "0476795b-e966-4cfe-a8af-ba1fc4610bda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:41.918425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918397    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0476795b-e966-4cfe-a8af-ba1fc4610bda" (UID: "0476795b-e966-4cfe-a8af-ba1fc4610bda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:41.918640 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918495    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476795b-e966-4cfe-a8af-ba1fc4610bda-istiod-ca-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:41.918640 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918515    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-credential-socket\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:41.918640 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918530    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-data\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:41.918640 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918544    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-socket\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:41.918640 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.918555    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-workload-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:41.920088 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.920060    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0476795b-e966-4cfe-a8af-ba1fc4610bda" (UID: "0476795b-e966-4cfe-a8af-ba1fc4610bda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:39:41.920463 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.920441    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-kube-api-access-zj767" (OuterVolumeSpecName: "kube-api-access-zj767") pod "0476795b-e966-4cfe-a8af-ba1fc4610bda" (UID: "0476795b-e966-4cfe-a8af-ba1fc4610bda"). InnerVolumeSpecName "kube-api-access-zj767". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:39:41.920547 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.920501    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0476795b-e966-4cfe-a8af-ba1fc4610bda" (UID: "0476795b-e966-4cfe-a8af-ba1fc4610bda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:41.920784 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:41.920754    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-token" (OuterVolumeSpecName: "istio-token") pod "0476795b-e966-4cfe-a8af-ba1fc4610bda" (UID: "0476795b-e966-4cfe-a8af-ba1fc4610bda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:39:42.019827 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.019796    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-podinfo\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:42.019827 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.019821    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-envoy\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:42.019973 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.019832    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-istio-token\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:42.019973 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.019844    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj767\" (UniqueName: \"kubernetes.io/projected/0476795b-e966-4cfe-a8af-ba1fc4610bda-kube-api-access-zj767\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:39:42.363483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.363452    2575 generic.go:356] "Generic (PLEG): container finished" podID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerID="4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b" exitCode=0
Sep 16 13:39:42.363678 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.363495    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"
Sep 16 13:39:42.363678 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.363501    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" event={"ID":"0476795b-e966-4cfe-a8af-ba1fc4610bda","Type":"ContainerDied","Data":"4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b"}
Sep 16 13:39:42.363678 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.363534    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd" event={"ID":"0476795b-e966-4cfe-a8af-ba1fc4610bda","Type":"ContainerDied","Data":"1de34b56c516d0a974736fe85e15ac6288abfd1b43dc017a5bccb2227806d751"}
Sep 16 13:39:42.363678 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.363554    2575 scope.go:117] "RemoveContainer" containerID="4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b"
Sep 16 13:39:42.373532 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.373515    2575 scope.go:117] "RemoveContainer" containerID="9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a"
Sep 16 13:39:42.381249 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.381233    2575 scope.go:117] "RemoveContainer" containerID="4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b"
Sep 16 13:39:42.381484 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:39:42.381469    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b\": container with ID starting with 4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b not found: ID does not exist" containerID="4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b"
Sep 16 13:39:42.381536 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.381491    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b"} err="failed to get container status \"4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b\": rpc error: code = NotFound desc = could not find container \"4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b\": container with ID starting with 4f0ccb0afccfe425337704885bb4e9b6b2edecffed61bb2cb6e26e50fa9a6b5b not found: ID does not exist"
Sep 16 13:39:42.381536 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.381505    2575 scope.go:117] "RemoveContainer" containerID="9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a"
Sep 16 13:39:42.381736 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:39:42.381721    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a\": container with ID starting with 9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a not found: ID does not exist" containerID="9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a"
Sep 16 13:39:42.381784 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.381742    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a"} err="failed to get container status \"9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a\": rpc error: code = NotFound desc = could not find container \"9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a\": container with ID starting with 9e6d1df2c46a0dd4341eb72254adead3f65f2e9569a1af898e9fce7c08d2310a not found: ID does not exist"
Sep 16 13:39:42.388006 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.387984    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"]
Sep 16 13:39:42.397808 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:42.397785    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-qcnqd"]
Sep 16 13:39:43.329036 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:43.329007    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" path="/var/lib/kubelet/pods/0476795b-e966-4cfe-a8af-ba1fc4610bda/volumes"
Sep 16 13:39:44.826447 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826414    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"]
Sep 16 13:39:44.826845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826784    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy"
Sep 16 13:39:44.826845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826796    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy"
Sep 16 13:39:44.826845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826807    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2795213f-9bcb-4cdf-80b4-17faceb9856c" containerName="main"
Sep 16 13:39:44.826845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826812    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2795213f-9bcb-4cdf-80b4-17faceb9856c" containerName="main"
Sep 16 13:39:44.826845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826824    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81acc17-e750-4b05-88e8-f7418db89084" containerName="main"
Sep 16 13:39:44.826845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826829    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81acc17-e750-4b05-88e8-f7418db89084" containerName="main"
Sep 16 13:39:44.826845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826836    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy"
Sep 16 13:39:44.826845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826841    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy"
Sep 16 13:39:44.827103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826903    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy"
Sep 16 13:39:44.827103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826912    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0476795b-e966-4cfe-a8af-ba1fc4610bda" containerName="istio-proxy"
Sep 16 13:39:44.827103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826919    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2795213f-9bcb-4cdf-80b4-17faceb9856c" containerName="main"
Sep 16 13:39:44.827103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.826924    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81acc17-e750-4b05-88e8-f7418db89084" containerName="main"
Sep 16 13:39:44.831428 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.831408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:44.835756 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.835734    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 13:39:44.835882 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.835773    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 13:39:44.835882 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.835779    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 13:39:44.835882 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.835837    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-2p8hv\""
Sep 16 13:39:44.835882 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.835848    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:39:44.837105 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.837075    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 16 13:39:44.860841 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.860810    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"]
Sep 16 13:39:44.874411 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.874387    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:44.941961 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.941941    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:44.942080 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.941966    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:44.942080 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.941990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:44.942080 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.942005    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d103bcec-ccf3-41cc-aff1-81f6d035cc52-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:44.942080 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.942067    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzw74\" (UniqueName: \"kubernetes.io/projected/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kube-api-access-kzw74\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:44.942306 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.942132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:44.942306 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.942173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:44.942306 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:44.942233    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.042953 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.042928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.042970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d103bcec-ccf3-41cc-aff1-81f6d035cc52-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzw74\" (UniqueName: \"kubernetes.io/projected/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kube-api-access-kzw74\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043142    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043411 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043411 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043515 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043462    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043606 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.043987 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.043969    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d103bcec-ccf3-41cc-aff1-81f6d035cc52-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.045784 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.045764    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.045953 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.045935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.052073 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.052054    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzw74\" (UniqueName: \"kubernetes.io/projected/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kube-api-access-kzw74\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.142532 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.142503    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:45.412031 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.411957    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:45.430486 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:45.430462    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"]
Sep 16 13:39:45.432062 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:39:45.432035    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd103bcec_ccf3_41cc_aff1_81f6d035cc52.slice/crio-0028758951df79460ef6bcb96250ff75c5d24eda4034dcdb456a1344774b1aae WatchSource:0}: Error finding container 0028758951df79460ef6bcb96250ff75c5d24eda4034dcdb456a1344774b1aae: Status 404 returned error can't find the container with id 0028758951df79460ef6bcb96250ff75c5d24eda4034dcdb456a1344774b1aae
Sep 16 13:39:46.221414 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:46.221374    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:46.385729 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:46.385688    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" event={"ID":"d103bcec-ccf3-41cc-aff1-81f6d035cc52","Type":"ContainerStarted","Data":"0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc"}
Sep 16 13:39:46.385922 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:46.385761    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" event={"ID":"d103bcec-ccf3-41cc-aff1-81f6d035cc52","Type":"ContainerStarted","Data":"0028758951df79460ef6bcb96250ff75c5d24eda4034dcdb456a1344774b1aae"}
Sep 16 13:39:46.534024 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:46.533882    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:48.228310 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:48.228269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:48.397548 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:48.397515    2575 generic.go:356] "Generic (PLEG): container finished" podID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerID="0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc" exitCode=0
Sep 16 13:39:48.397548 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:48.397548    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" event={"ID":"d103bcec-ccf3-41cc-aff1-81f6d035cc52","Type":"ContainerDied","Data":"0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc"}
Sep 16 13:39:49.404199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:49.404159    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" event={"ID":"d103bcec-ccf3-41cc-aff1-81f6d035cc52","Type":"ContainerStarted","Data":"a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57"}
Sep 16 13:39:49.461805 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:49.461751    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podStartSLOduration=3.161523423 podStartE2EDuration="5.461735845s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.300212422s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:39:49.458246539 +0000 UTC m=+3110.639389191" watchObservedRunningTime="2026-09-16 13:39:49.461735845 +0000 UTC m=+3110.642878498"
Sep 16 13:39:50.143110 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:50.143077    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:50.722115 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:50.722084    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:53.227984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:53.227898    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:55.142719 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:55.142678    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:39:55.144350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:55.144320    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:39:56.511550 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:56.511512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:57.455789 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:57.455750    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:57.578338 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:57.578304    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:59.538266 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:59.538232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:59.636854 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:39:59.636811    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:05.143389 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:05.143335    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:40:09.220745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.220707    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:09.321449 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.321414    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"]
Sep 16 13:40:09.326679 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.326638    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.329998 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.329962    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 13:40:09.330462 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.330427    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 13:40:09.330584 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.330336    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 13:40:09.330756 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.330349    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:40:09.330832 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.330408    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 13:40:09.330832 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.330380    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-zsl4k\""
Sep 16 13:40:09.338791 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.338767    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"]
Sep 16 13:40:09.346338 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.346307    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.346456 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.346351    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6qh\" (UniqueName: \"kubernetes.io/projected/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kube-api-access-9f6qh\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.346456 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.346400    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56ca48dd-392b-4dce-b0a4-c9286b7652f2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.346456 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.346439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.346669 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.346469    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.346669 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.346516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.346669 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.346568    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.346669 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.346597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.348634 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.348608    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:09.447883 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.447851    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.447887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.447915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.447991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.448015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6qh\" (UniqueName: \"kubernetes.io/projected/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kube-api-access-9f6qh\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.448041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56ca48dd-392b-4dce-b0a4-c9286b7652f2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448310 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.448076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448310 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.448274    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448425 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.448318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.448535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.448943 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.448916    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56ca48dd-392b-4dce-b0a4-c9286b7652f2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.449038 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.448690    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.449038 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.448638    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.450349 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.450331    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.450783 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.450763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.457222 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.457168    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6qh\" (UniqueName: \"kubernetes.io/projected/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kube-api-access-9f6qh\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.581199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.581163    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:09.640772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.640738    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:09.928918 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:09.928878    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"]
Sep 16 13:40:09.929997 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:40:09.929968    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ca48dd_392b_4dce_b0a4_c9286b7652f2.slice/crio-cd47cd1f3b368c4e5157999bb57ad48755bf3a46316130fe44fcb857d8ceef7b WatchSource:0}: Error finding container cd47cd1f3b368c4e5157999bb57ad48755bf3a46316130fe44fcb857d8ceef7b: Status 404 returned error can't find the container with id cd47cd1f3b368c4e5157999bb57ad48755bf3a46316130fe44fcb857d8ceef7b
Sep 16 13:40:10.525561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:10.525522    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" event={"ID":"56ca48dd-392b-4dce-b0a4-c9286b7652f2","Type":"ContainerStarted","Data":"e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f"}
Sep 16 13:40:10.526190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:10.526161    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" event={"ID":"56ca48dd-392b-4dce-b0a4-c9286b7652f2","Type":"ContainerStarted","Data":"cd47cd1f3b368c4e5157999bb57ad48755bf3a46316130fe44fcb857d8ceef7b"}
Sep 16 13:40:11.658994 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:11.658953    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:11.722599 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:11.722559    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:13.548410 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:13.548377    2575 generic.go:356] "Generic (PLEG): container finished" podID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerID="e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f" exitCode=0
Sep 16 13:40:13.548799 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:13.548460    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" event={"ID":"56ca48dd-392b-4dce-b0a4-c9286b7652f2","Type":"ContainerDied","Data":"e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f"}
Sep 16 13:40:14.556627 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:14.556589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" event={"ID":"56ca48dd-392b-4dce-b0a4-c9286b7652f2","Type":"ContainerStarted","Data":"f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c"}
Sep 16 13:40:14.596622 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:14.596563    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podStartSLOduration=2.7110816140000003 podStartE2EDuration="5.596544913s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.885463299s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:14.593355827 +0000 UTC m=+3135.774498480" watchObservedRunningTime="2026-09-16 13:40:14.596544913 +0000 UTC m=+3135.777687565"
Sep 16 13:40:14.641816 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:14.641783    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:15.143381 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:15.143342    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:40:19.641700 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:19.641658    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:40:19.643712 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:19.643682    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 13:40:25.143342 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:25.143248    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:40:25.357085 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:25.357045    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:29.642021 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:29.641976    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 13:40:35.143339 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:35.143294    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:40:39.641746 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:39.641695    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 13:40:45.143077 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:45.143035    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:40:49.642052 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:49.642012    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 13:40:55.143846 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:55.143797    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:40:59.641985 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:40:59.641934    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 13:41:05.143488 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:05.143444    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:41:09.642120 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:09.642064    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 13:41:14.822379 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:14.822346    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:15.143059 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:15.143019    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:41:19.641367 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:19.641317    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 13:41:25.143368 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:25.143325    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:41:29.651151 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:29.651115    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:41:29.659538 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:29.659506    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:41:35.143108 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:35.143069    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:41:45.153167 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:45.153133    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:41:45.162797 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:45.162768    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:41:50.066660 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:50.066596    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:52.872987 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:52.872912    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"]
Sep 16 13:41:52.877415 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:52.877383    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:52.947530 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:41:52.947493    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:04.168011 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:04.167964    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:04.168549 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:04.168336    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"]
Sep 16 13:42:04.263683 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:04.263627    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:07.896249 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:07.896190    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" containerID="cri-o://a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57" gracePeriod=45
Sep 16 13:42:08.154132 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.154024    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 13:42:08.651750 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.651726    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:42:08.741912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.741837    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d103bcec-ccf3-41cc-aff1-81f6d035cc52-cabundle-cert\") pod \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") "
Sep 16 13:42:08.741912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.741878    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-model-cache\") pod \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") "
Sep 16 13:42:08.741912 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.741908    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kserve-provision-location\") pod \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") "
Sep 16 13:42:08.742192 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.741945    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tmp-dir\") pod \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") "
Sep 16 13:42:08.742192 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.741968    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tls-certs\") pod \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") "
Sep 16 13:42:08.742192 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.742014    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzw74\" (UniqueName: \"kubernetes.io/projected/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kube-api-access-kzw74\") pod \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") "
Sep 16 13:42:08.742192 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.742141    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-model-cache" (OuterVolumeSpecName: "model-cache") pod "d103bcec-ccf3-41cc-aff1-81f6d035cc52" (UID: "d103bcec-ccf3-41cc-aff1-81f6d035cc52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:08.742192 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.742150    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-home\") pod \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") "
Sep 16 13:42:08.742465 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.742221    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-dshm\") pod \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\" (UID: \"d103bcec-ccf3-41cc-aff1-81f6d035cc52\") "
Sep 16 13:42:08.742465 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.742240    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d103bcec-ccf3-41cc-aff1-81f6d035cc52-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d103bcec-ccf3-41cc-aff1-81f6d035cc52" (UID: "d103bcec-ccf3-41cc-aff1-81f6d035cc52"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:42:08.742576 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.742493    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d103bcec-ccf3-41cc-aff1-81f6d035cc52-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:08.742576 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.742512    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:08.742762 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.742742    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-home" (OuterVolumeSpecName: "home") pod "d103bcec-ccf3-41cc-aff1-81f6d035cc52" (UID: "d103bcec-ccf3-41cc-aff1-81f6d035cc52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:08.744571 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.744543    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-dshm" (OuterVolumeSpecName: "dshm") pod "d103bcec-ccf3-41cc-aff1-81f6d035cc52" (UID: "d103bcec-ccf3-41cc-aff1-81f6d035cc52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:08.744825 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.744801    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d103bcec-ccf3-41cc-aff1-81f6d035cc52" (UID: "d103bcec-ccf3-41cc-aff1-81f6d035cc52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:42:08.744825 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.744811    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kube-api-access-kzw74" (OuterVolumeSpecName: "kube-api-access-kzw74") pod "d103bcec-ccf3-41cc-aff1-81f6d035cc52" (UID: "d103bcec-ccf3-41cc-aff1-81f6d035cc52"). InnerVolumeSpecName "kube-api-access-kzw74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:42:08.755214 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.755190    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d103bcec-ccf3-41cc-aff1-81f6d035cc52" (UID: "d103bcec-ccf3-41cc-aff1-81f6d035cc52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:08.763317 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.763296    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d103bcec-ccf3-41cc-aff1-81f6d035cc52" (UID: "d103bcec-ccf3-41cc-aff1-81f6d035cc52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:08.843055 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.843030    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:08.843055 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.843050    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:08.843191 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.843061    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d103bcec-ccf3-41cc-aff1-81f6d035cc52-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:08.843191 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.843070    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzw74\" (UniqueName: \"kubernetes.io/projected/d103bcec-ccf3-41cc-aff1-81f6d035cc52-kube-api-access-kzw74\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:08.843191 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.843081    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:08.843191 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:08.843093    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d103bcec-ccf3-41cc-aff1-81f6d035cc52-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:09.241162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.241129    2575 generic.go:356] "Generic (PLEG): container finished" podID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerID="a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57" exitCode=0
Sep 16 13:42:09.241571 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.241170    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"
Sep 16 13:42:09.241571 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.241206    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" event={"ID":"d103bcec-ccf3-41cc-aff1-81f6d035cc52","Type":"ContainerDied","Data":"a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57"}
Sep 16 13:42:09.241571 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.241257    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf" event={"ID":"d103bcec-ccf3-41cc-aff1-81f6d035cc52","Type":"ContainerDied","Data":"0028758951df79460ef6bcb96250ff75c5d24eda4034dcdb456a1344774b1aae"}
Sep 16 13:42:09.241571 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.241279    2575 scope.go:117] "RemoveContainer" containerID="a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57"
Sep 16 13:42:09.251329 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.251315    2575 scope.go:117] "RemoveContainer" containerID="0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc"
Sep 16 13:42:09.259431 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.259416    2575 scope.go:117] "RemoveContainer" containerID="a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57"
Sep 16 13:42:09.259710 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:42:09.259691    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57\": container with ID starting with a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57 not found: ID does not exist" containerID="a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57"
Sep 16 13:42:09.259785 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.259718    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57"} err="failed to get container status \"a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57\": rpc error: code = NotFound desc = could not find container \"a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57\": container with ID starting with a407015560f8d16abf950dc1185f39c8ab9f850c58490462c18851823b4ccb57 not found: ID does not exist"
Sep 16 13:42:09.259785 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.259736    2575 scope.go:117] "RemoveContainer" containerID="0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc"
Sep 16 13:42:09.259968 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:42:09.259950    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc\": container with ID starting with 0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc not found: ID does not exist" containerID="0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc"
Sep 16 13:42:09.260013 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.259974    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc"} err="failed to get container status \"0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc\": rpc error: code = NotFound desc = could not find container \"0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc\": container with ID starting with 0682d43931de38d37c06d81673932be584074620662f6e5d439d1ddc0e1c15dc not found: ID does not exist"
Sep 16 13:42:09.264384 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.264364    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"]
Sep 16 13:42:09.268424 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.268400    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5c56c467d8tndcf"]
Sep 16 13:42:09.329178 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:09.329155    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" path="/var/lib/kubelet/pods/d103bcec-ccf3-41cc-aff1-81f6d035cc52/volumes"
Sep 16 13:42:19.192022 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.191948    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main" containerID="cri-o://f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c" gracePeriod=45
Sep 16 13:42:19.652076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.652041    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 13:42:19.833602 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.833576    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:42:19.930489 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930409    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-home\") pod \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") "
Sep 16 13:42:19.930489 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930459    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f6qh\" (UniqueName: \"kubernetes.io/projected/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kube-api-access-9f6qh\") pod \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") "
Sep 16 13:42:19.930745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930494    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-model-cache\") pod \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") "
Sep 16 13:42:19.930745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930517    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tls-certs\") pod \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") "
Sep 16 13:42:19.930745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930566    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-dshm\") pod \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") "
Sep 16 13:42:19.930745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930665    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56ca48dd-392b-4dce-b0a4-c9286b7652f2-cabundle-cert\") pod \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") "
Sep 16 13:42:19.930745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930704    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tmp-dir\") pod \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") "
Sep 16 13:42:19.930745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930737    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kserve-provision-location\") pod \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\" (UID: \"56ca48dd-392b-4dce-b0a4-c9286b7652f2\") "
Sep 16 13:42:19.931057 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930834    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "56ca48dd-392b-4dce-b0a4-c9286b7652f2" (UID: "56ca48dd-392b-4dce-b0a4-c9286b7652f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:19.931057 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.930945    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-home" (OuterVolumeSpecName: "home") pod "56ca48dd-392b-4dce-b0a4-c9286b7652f2" (UID: "56ca48dd-392b-4dce-b0a4-c9286b7652f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:19.931145 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.931066    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:19.931145 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.931088    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:19.931678 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.931629    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ca48dd-392b-4dce-b0a4-c9286b7652f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56ca48dd-392b-4dce-b0a4-c9286b7652f2" (UID: "56ca48dd-392b-4dce-b0a4-c9286b7652f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:42:19.933106 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.933085    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-dshm" (OuterVolumeSpecName: "dshm") pod "56ca48dd-392b-4dce-b0a4-c9286b7652f2" (UID: "56ca48dd-392b-4dce-b0a4-c9286b7652f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:19.933313 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.933285    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kube-api-access-9f6qh" (OuterVolumeSpecName: "kube-api-access-9f6qh") pod "56ca48dd-392b-4dce-b0a4-c9286b7652f2" (UID: "56ca48dd-392b-4dce-b0a4-c9286b7652f2"). InnerVolumeSpecName "kube-api-access-9f6qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:42:19.933438 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.933328    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56ca48dd-392b-4dce-b0a4-c9286b7652f2" (UID: "56ca48dd-392b-4dce-b0a4-c9286b7652f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:42:19.943239 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.943203    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56ca48dd-392b-4dce-b0a4-c9286b7652f2" (UID: "56ca48dd-392b-4dce-b0a4-c9286b7652f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:19.953923 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:19.953898    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56ca48dd-392b-4dce-b0a4-c9286b7652f2" (UID: "56ca48dd-392b-4dce-b0a4-c9286b7652f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:42:20.032339 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.032316    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56ca48dd-392b-4dce-b0a4-c9286b7652f2-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:20.032339 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.032336    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:20.032467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.032345    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:20.032467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.032353    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6qh\" (UniqueName: \"kubernetes.io/projected/56ca48dd-392b-4dce-b0a4-c9286b7652f2-kube-api-access-9f6qh\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:20.032467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.032361    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ca48dd-392b-4dce-b0a4-c9286b7652f2-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:20.032467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.032370    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ca48dd-392b-4dce-b0a4-c9286b7652f2-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:42:20.298385 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.298307    2575 generic.go:356] "Generic (PLEG): container finished" podID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerID="f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c" exitCode=0
Sep 16 13:42:20.298385 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.298363    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"
Sep 16 13:42:20.298385 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.298363    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" event={"ID":"56ca48dd-392b-4dce-b0a4-c9286b7652f2","Type":"ContainerDied","Data":"f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c"}
Sep 16 13:42:20.298902 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.298427    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28" event={"ID":"56ca48dd-392b-4dce-b0a4-c9286b7652f2","Type":"ContainerDied","Data":"cd47cd1f3b368c4e5157999bb57ad48755bf3a46316130fe44fcb857d8ceef7b"}
Sep 16 13:42:20.298902 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.298458    2575 scope.go:117] "RemoveContainer" containerID="f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c"
Sep 16 13:42:20.307897 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.307883    2575 scope.go:117] "RemoveContainer" containerID="e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f"
Sep 16 13:42:20.316464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.316443    2575 scope.go:117] "RemoveContainer" containerID="f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c"
Sep 16 13:42:20.316745 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:42:20.316729    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c\": container with ID starting with f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c not found: ID does not exist" containerID="f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c"
Sep 16 13:42:20.316800 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.316752    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c"} err="failed to get container status \"f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c\": rpc error: code = NotFound desc = could not find container \"f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c\": container with ID starting with f8e73ffd5ce824da42f56cd4c805e779ddab130d102806c24bb3d38bd05a595c not found: ID does not exist"
Sep 16 13:42:20.316800 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.316768    2575 scope.go:117] "RemoveContainer" containerID="e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f"
Sep 16 13:42:20.316988 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:42:20.316971    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f\": container with ID starting with e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f not found: ID does not exist" containerID="e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f"
Sep 16 13:42:20.317029 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.316995    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f"} err="failed to get container status \"e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f\": rpc error: code = NotFound desc = could not find container \"e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f\": container with ID starting with e08dc8c7b8dee8f05d9255e6a796adfb951690abe3d798ef527b36d90468070f not found: ID does not exist"
Sep 16 13:42:20.322376 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.322354    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"]
Sep 16 13:42:20.325942 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:20.325919    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79745476b4-kzl28"]
Sep 16 13:42:21.327678 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:21.327627    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" path="/var/lib/kubelet/pods/56ca48dd-392b-4dce-b0a4-c9286b7652f2/volumes"
Sep 16 13:42:42.995773 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:42.995740    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:43.086791 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:43.086761    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:49.416310 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:49.416265    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:54.273623 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:54.273583    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:54.372446 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:54.372413    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:55.904008 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:55.903965    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:56.005723 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:42:56.005690    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:00.431209 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:00.431168    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:06.277147 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:06.277106    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:06.348154 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:06.348118    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:11.499216 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499184    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"]
Sep 16 13:43:11.499611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499530    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main"
Sep 16 13:43:11.499611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499540    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main"
Sep 16 13:43:11.499611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499555    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main"
Sep 16 13:43:11.499611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499561    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main"
Sep 16 13:43:11.499611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499568    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="storage-initializer"
Sep 16 13:43:11.499611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499574    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="storage-initializer"
Sep 16 13:43:11.499611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499584    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="storage-initializer"
Sep 16 13:43:11.499611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499589    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="storage-initializer"
Sep 16 13:43:11.499928 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499658    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ca48dd-392b-4dce-b0a4-c9286b7652f2" containerName="main"
Sep 16 13:43:11.499928 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.499668    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d103bcec-ccf3-41cc-aff1-81f6d035cc52" containerName="main"
Sep 16 13:43:11.502842 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.502824    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.505791 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.505757    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 13:43:11.505918 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.505806    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 13:43:11.505918 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.505856    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 13:43:11.505918 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.505860    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-c99cs\""
Sep 16 13:43:11.517603 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.517571    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"]
Sep 16 13:43:11.535366 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.535335    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:11.618861 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.618826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.619029 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.618868    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.619029 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.618892    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.619029 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.618947    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.619029 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.619008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.619201 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.619031    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.619201 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.619050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4w7k\" (UniqueName: \"kubernetes.io/projected/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kube-api-access-p4w7k\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.719843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.719808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.719843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.719851    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4w7k\" (UniqueName: \"kubernetes.io/projected/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kube-api-access-p4w7k\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.720087 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.719905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.720087 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.719947    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.720087 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.719986    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.720087 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.720016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.720087 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.720059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.720504 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.720479    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.720886 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.720858    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.720973 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.720883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.720973 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.720955    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.723009 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.722989    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.723419 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.723399    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.732372 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.732347    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4w7k\" (UniqueName: \"kubernetes.io/projected/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kube-api-access-p4w7k\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:11.765254 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.765192    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:11.812953 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:11.812929    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:12.094721 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:12.094678    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"]
Sep 16 13:43:12.136929 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:12.136907    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:43:12.564065 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:12.564035    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" event={"ID":"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713","Type":"ContainerStarted","Data":"d90dd2e069672e7a49138b1406ef6d256d6715af49a152a92f1e28b8d79ba093"}
Sep 16 13:43:17.751697 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.751661    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:17.934032 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.933995    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"]
Sep 16 13:43:17.939196 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.939166    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:17.944229 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.944089    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-8mbht\""
Sep 16 13:43:17.944229 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.944108    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 13:43:17.944450 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.944298    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 13:43:17.944450 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.944380    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 13:43:17.950799 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.950773    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-llk2x\""
Sep 16 13:43:17.961473 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.961446    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"]
Sep 16 13:43:17.992078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:17.992049    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:18.076074 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:18.075977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxqp\" (UniqueName: \"kubernetes.io/projected/b27722f3-d2eb-4597-ac9b-683458475e53-kube-api-access-dqxqp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6\" (UID: \"b27722f3-d2eb-4597-ac9b-683458475e53\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:18.076074 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:18.076046    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27722f3-d2eb-4597-ac9b-683458475e53-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6\" (UID: \"b27722f3-d2eb-4597-ac9b-683458475e53\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:18.177584 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:18.177552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxqp\" (UniqueName: \"kubernetes.io/projected/b27722f3-d2eb-4597-ac9b-683458475e53-kube-api-access-dqxqp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6\" (UID: \"b27722f3-d2eb-4597-ac9b-683458475e53\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:18.177781 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:18.177603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27722f3-d2eb-4597-ac9b-683458475e53-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6\" (UID: \"b27722f3-d2eb-4597-ac9b-683458475e53\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:18.180506 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:18.180485    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27722f3-d2eb-4597-ac9b-683458475e53-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6\" (UID: \"b27722f3-d2eb-4597-ac9b-683458475e53\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:18.187041 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:18.187019    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxqp\" (UniqueName: \"kubernetes.io/projected/b27722f3-d2eb-4597-ac9b-683458475e53-kube-api-access-dqxqp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6\" (UID: \"b27722f3-d2eb-4597-ac9b-683458475e53\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:18.252763 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:18.252733    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:18.827971 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:18.827947    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"]
Sep 16 13:43:18.829593 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:43:18.829556    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27722f3_d2eb_4597_ac9b_683458475e53.slice/crio-5e69c7cf24366125ea1f1b8acf0902d6ea918f71d356d21a7cdec730021a5f5c WatchSource:0}: Error finding container 5e69c7cf24366125ea1f1b8acf0902d6ea918f71d356d21a7cdec730021a5f5c: Status 404 returned error can't find the container with id 5e69c7cf24366125ea1f1b8acf0902d6ea918f71d356d21a7cdec730021a5f5c
Sep 16 13:43:19.603142 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:19.603098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" event={"ID":"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713","Type":"ContainerStarted","Data":"39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4"}
Sep 16 13:43:19.604662 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:19.604620    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" event={"ID":"b27722f3-d2eb-4597-ac9b-683458475e53","Type":"ContainerStarted","Data":"f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6"}
Sep 16 13:43:19.604808 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:19.604684    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" event={"ID":"b27722f3-d2eb-4597-ac9b-683458475e53","Type":"ContainerStarted","Data":"5e69c7cf24366125ea1f1b8acf0902d6ea918f71d356d21a7cdec730021a5f5c"}
Sep 16 13:43:19.604808 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:19.604725    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:19.650190 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:19.650141    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" podStartSLOduration=2.650123482 podStartE2EDuration="2.650123482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:19.648726099 +0000 UTC m=+3320.829868749" watchObservedRunningTime="2026-09-16 13:43:19.650123482 +0000 UTC m=+3320.831266164"
Sep 16 13:43:20.610934 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:20.610905    2575 generic.go:356] "Generic (PLEG): container finished" podID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerID="39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4" exitCode=0
Sep 16 13:43:20.611398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:20.610993    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" event={"ID":"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713","Type":"ContainerDied","Data":"39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4"}
Sep 16 13:43:21.619841 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:21.619807    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" event={"ID":"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713","Type":"ContainerStarted","Data":"711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea"}
Sep 16 13:43:21.620194 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:21.619868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" event={"ID":"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713","Type":"ContainerStarted","Data":"8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071"}
Sep 16 13:43:21.654526 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:21.654475    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podStartSLOduration=2.587995543 podStartE2EDuration="10.654457715s" totalImagesPullingTime="6.779675588s" totalInitContainerRuntime="1.286786584s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:21.651168276 +0000 UTC m=+3322.832310929" watchObservedRunningTime="2026-09-16 13:43:21.654457715 +0000 UTC m=+3322.835600370"
Sep 16 13:43:21.813832 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:21.813799    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:21.813996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:21.813876    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:43:21.815551 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:21.815516    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:43:24.823510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:24.823476    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:30.340288 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:30.340255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:31.813612 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:31.813567    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:43:41.813527 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:41.813482    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:43:50.612923 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:50.612891    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:43:50.968165 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:50.968087    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:51.814311 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:43:51.814270    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:44:01.813703 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:01.813634    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:44:11.813609 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:11.813562    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:44:21.814309 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:21.814260    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:44:31.813854 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:31.813809    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:44:39.618747 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:39.618710    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"]
Sep 16 13:44:39.643028 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:39.642996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:44:39.739930 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:39.739893    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:44:40.613375 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:40.613342    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:44:41.814405 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:41.814357    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:44:50.613183 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:50.613106    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:44:51.814094 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:51.814049    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:44:54.648796 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:44:54.648739    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" containerName="main" containerID="cri-o://f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6" gracePeriod=45
Sep 16 13:45:00.612972 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:00.612927    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:45:00.613487 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:00.613065    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:45:01.814198 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:01.814148    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:45:10.613275 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:10.613241    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:45:11.824257 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:11.824225    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:45:11.832017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:11.831996    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:45:16.285175 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:16.285137    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:16.292682 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:16.292656    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"]
Sep 16 13:45:16.292980 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:16.292926    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="modelcar" containerID="cri-o://711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea" gracePeriod=60
Sep 16 13:45:16.366372 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:16.366341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:17.266857 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:17.266830    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52_a27f3bc1-7bbc-4206-9fbc-0880cc5e1713/modelcar/0.log"
Sep 16 13:45:17.267503 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:17.267480    2575 generic.go:356] "Generic (PLEG): container finished" podID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerID="711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea" exitCode=143
Sep 16 13:45:17.267595 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:17.267519    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" event={"ID":"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713","Type":"ContainerDied","Data":"711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea"}
Sep 16 13:45:18.855604 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:18.855558    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:20.612468 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:20.612436    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 13:45:24.909493 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:24.909438    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:45:24.950558 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:24.950531    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqxqp\" (UniqueName: \"kubernetes.io/projected/b27722f3-d2eb-4597-ac9b-683458475e53-kube-api-access-dqxqp\") pod \"b27722f3-d2eb-4597-ac9b-683458475e53\" (UID: \"b27722f3-d2eb-4597-ac9b-683458475e53\") "
Sep 16 13:45:24.950697 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:24.950633    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27722f3-d2eb-4597-ac9b-683458475e53-tls-certs\") pod \"b27722f3-d2eb-4597-ac9b-683458475e53\" (UID: \"b27722f3-d2eb-4597-ac9b-683458475e53\") "
Sep 16 13:45:24.952837 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:24.952805    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27722f3-d2eb-4597-ac9b-683458475e53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b27722f3-d2eb-4597-ac9b-683458475e53" (UID: "b27722f3-d2eb-4597-ac9b-683458475e53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:24.952948 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:24.952860    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27722f3-d2eb-4597-ac9b-683458475e53-kube-api-access-dqxqp" (OuterVolumeSpecName: "kube-api-access-dqxqp") pod "b27722f3-d2eb-4597-ac9b-683458475e53" (UID: "b27722f3-d2eb-4597-ac9b-683458475e53"). InnerVolumeSpecName "kube-api-access-dqxqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:25.051307 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.051279    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxqp\" (UniqueName: \"kubernetes.io/projected/b27722f3-d2eb-4597-ac9b-683458475e53-kube-api-access-dqxqp\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:25.051307 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.051302    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27722f3-d2eb-4597-ac9b-683458475e53-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:25.310131 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.310047    2575 generic.go:356] "Generic (PLEG): container finished" podID="b27722f3-d2eb-4597-ac9b-683458475e53" containerID="f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6" exitCode=0
Sep 16 13:45:25.310131 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.310091    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"
Sep 16 13:45:25.310131 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.310117    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" event={"ID":"b27722f3-d2eb-4597-ac9b-683458475e53","Type":"ContainerDied","Data":"f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6"}
Sep 16 13:45:25.310335 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.310168    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6" event={"ID":"b27722f3-d2eb-4597-ac9b-683458475e53","Type":"ContainerDied","Data":"5e69c7cf24366125ea1f1b8acf0902d6ea918f71d356d21a7cdec730021a5f5c"}
Sep 16 13:45:25.310335 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.310212    2575 scope.go:117] "RemoveContainer" containerID="f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6"
Sep 16 13:45:25.319592 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.319576    2575 scope.go:117] "RemoveContainer" containerID="f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6"
Sep 16 13:45:25.319876 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:45:25.319859    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6\": container with ID starting with f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6 not found: ID does not exist" containerID="f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6"
Sep 16 13:45:25.319940 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.319885    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6"} err="failed to get container status \"f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6\": rpc error: code = NotFound desc = could not find container \"f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6\": container with ID starting with f00558e7518e355bdc755d20b7610e985670797691d4806f62f0e2db6731b9d6 not found: ID does not exist"
Sep 16 13:45:25.332678 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.332619    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"]
Sep 16 13:45:25.337770 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:25.337747    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6nbx6"]
Sep 16 13:45:27.329039 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:27.329000    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" path="/var/lib/kubelet/pods/b27722f3-d2eb-4597-ac9b-683458475e53/volumes"
Sep 16 13:45:29.729612 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:29.729580    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.842979 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:29.842949    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.942682 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:29.942632    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:31.315000 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:31.314960    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" containerID="cri-o://8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071" gracePeriod=45
Sep 16 13:45:31.825824 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:31.825792    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 13:45:32.009772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.009746    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:32.091977 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.091956    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52_a27f3bc1-7bbc-4206-9fbc-0880cc5e1713/modelcar/0.log"
Sep 16 13:45:32.097571 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.097551    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:45:32.109521 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.109496    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:32.208983 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.208956    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-model-cache\") pod \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") "
Sep 16 13:45:32.209129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209007    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:32.209129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209040    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-dshm\") pod \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") "
Sep 16 13:45:32.209129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209086    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tls-certs\") pod \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") "
Sep 16 13:45:32.209129 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209112    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-home\") pod \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") "
Sep 16 13:45:32.209362 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209202    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-model-cache" (OuterVolumeSpecName: "model-cache") pod "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" (UID: "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:32.209362 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209249    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tmp-dir\") pod \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") "
Sep 16 13:45:32.209362 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209344    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kserve-provision-location\") pod \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") "
Sep 16 13:45:32.209515 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209381    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4w7k\" (UniqueName: \"kubernetes.io/projected/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kube-api-access-p4w7k\") pod \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\" (UID: \"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713\") "
Sep 16 13:45:32.209672 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209631    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:32.209765 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209676    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" (UID: "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:32.209883 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.209863    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-home" (OuterVolumeSpecName: "home") pod "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" (UID: "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:32.211992 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.211969    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-dshm" (OuterVolumeSpecName: "dshm") pod "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" (UID: "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:32.212138 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.212117    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" (UID: "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:32.212192 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.212132    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kube-api-access-p4w7k" (OuterVolumeSpecName: "kube-api-access-p4w7k") pod "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" (UID: "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713"). InnerVolumeSpecName "kube-api-access-p4w7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:32.220599 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.220576    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" (UID: "a27f3bc1-7bbc-4206-9fbc-0880cc5e1713"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:32.310637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.310612    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:32.310637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.310633    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:32.310839 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.310666    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:32.310839 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.310685    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:32.310839 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.310697    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:32.310839 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.310709    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4w7k\" (UniqueName: \"kubernetes.io/projected/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713-kube-api-access-p4w7k\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:32.348283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.348228    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52_a27f3bc1-7bbc-4206-9fbc-0880cc5e1713/modelcar/0.log"
Sep 16 13:45:32.348982 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.348963    2575 generic.go:356] "Generic (PLEG): container finished" podID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerID="8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071" exitCode=0
Sep 16 13:45:32.349035 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.348996    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" event={"ID":"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713","Type":"ContainerDied","Data":"8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071"}
Sep 16 13:45:32.349078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.349030    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"
Sep 16 13:45:32.349078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.349040    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52" event={"ID":"a27f3bc1-7bbc-4206-9fbc-0880cc5e1713","Type":"ContainerDied","Data":"d90dd2e069672e7a49138b1406ef6d256d6715af49a152a92f1e28b8d79ba093"}
Sep 16 13:45:32.349078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.349066    2575 scope.go:117] "RemoveContainer" containerID="711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea"
Sep 16 13:45:32.358745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.358730    2575 scope.go:117] "RemoveContainer" containerID="8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071"
Sep 16 13:45:32.366762 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.366746    2575 scope.go:117] "RemoveContainer" containerID="39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4"
Sep 16 13:45:32.374237 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.374212    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"]
Sep 16 13:45:32.374996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.374984    2575 scope.go:117] "RemoveContainer" containerID="711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea"
Sep 16 13:45:32.375257 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:45:32.375241    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea\": container with ID starting with 711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea not found: ID does not exist" containerID="711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea"
Sep 16 13:45:32.375297 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.375265    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea"} err="failed to get container status \"711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea\": rpc error: code = NotFound desc = could not find container \"711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea\": container with ID starting with 711ceccbdcf53613301f684e704e43ac7aae011375e848b991f6e7e3dd3e0fea not found: ID does not exist"
Sep 16 13:45:32.375297 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.375286    2575 scope.go:117] "RemoveContainer" containerID="8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071"
Sep 16 13:45:32.375521 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:45:32.375498    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071\": container with ID starting with 8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071 not found: ID does not exist" containerID="8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071"
Sep 16 13:45:32.375591 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.375558    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071"} err="failed to get container status \"8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071\": rpc error: code = NotFound desc = could not find container \"8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071\": container with ID starting with 8ba03681004f14b496b6332957a8878e0fb2d92c25555ade379d2b1a0eba4071 not found: ID does not exist"
Sep 16 13:45:32.375591 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.375585    2575 scope.go:117] "RemoveContainer" containerID="39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4"
Sep 16 13:45:32.375939 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:45:32.375910    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4\": container with ID starting with 39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4 not found: ID does not exist" containerID="39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4"
Sep 16 13:45:32.376061 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.375945    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4"} err="failed to get container status \"39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4\": rpc error: code = NotFound desc = could not find container \"39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4\": container with ID starting with 39a52cf971f2e7c1b5b117f25d24d30b182dd61978967d59ebbcea20c5db02f4 not found: ID does not exist"
Sep 16 13:45:32.378443 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:32.378422    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-t4z52"]
Sep 16 13:45:33.328877 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:33.328844    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" path="/var/lib/kubelet/pods/a27f3bc1-7bbc-4206-9fbc-0880cc5e1713/volumes"
Sep 16 13:45:36.185510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:36.185478    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:38.050285 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:38.050248    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:38.152576 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:38.152540    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:42.023517 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:42.023481    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:42.124297 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:42.124270    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:45.638626 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.638594    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"]
Sep 16 13:45:45.639048 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.638974    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main"
Sep 16 13:45:45.639048 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.638986    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main"
Sep 16 13:45:45.639048 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.638995    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="modelcar"
Sep 16 13:45:45.639048 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.639000    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="modelcar"
Sep 16 13:45:45.639048 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.639008    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" containerName="main"
Sep 16 13:45:45.639048 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.639013    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" containerName="main"
Sep 16 13:45:45.639048 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.639025    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="modelcar-init"
Sep 16 13:45:45.639048 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.639030    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="modelcar-init"
Sep 16 13:45:45.639332 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.639089    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="modelcar"
Sep 16 13:45:45.639332 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.639099    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b27722f3-d2eb-4597-ac9b-683458475e53" containerName="main"
Sep 16 13:45:45.639332 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.639106    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27f3bc1-7bbc-4206-9fbc-0880cc5e1713" containerName="main"
Sep 16 13:45:45.642097 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.642080    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.645269 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.645235    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 16 13:45:45.645395 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.645319    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 13:45:45.645561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.645547    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 13:45:45.645989 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.645965    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 13:45:45.646101 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.646005    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-jxglr\""
Sep 16 13:45:45.659060 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.659038    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"]
Sep 16 13:45:45.664549 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.664519    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:45.710011 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.709986    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.710119 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.710025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-home\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.710119 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.710045    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab93e788-1429-4b26-9342-4b6ff21e692f-tls-certs\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.710119 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.710064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-model-cache\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.710119 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.710097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.710281 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.710137    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7mcn\" (UniqueName: \"kubernetes.io/projected/ab93e788-1429-4b26-9342-4b6ff21e692f-kube-api-access-f7mcn\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.710281 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.710179    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-dshm\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.810915 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.810889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811036 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.810929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-home\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811036 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.810949    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab93e788-1429-4b26-9342-4b6ff21e692f-tls-certs\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811036 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.810966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-model-cache\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811218 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.811071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811218 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.811103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7mcn\" (UniqueName: \"kubernetes.io/projected/ab93e788-1429-4b26-9342-4b6ff21e692f-kube-api-access-f7mcn\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811218 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.811157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-dshm\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811373 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.811260    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811447 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.811426    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-model-cache\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811509 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.811448    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-home\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.811564 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.811499    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.813489 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.813467    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-dshm\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.813637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.813619    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab93e788-1429-4b26-9342-4b6ff21e692f-tls-certs\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.820587 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.820560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7mcn\" (UniqueName: \"kubernetes.io/projected/ab93e788-1429-4b26-9342-4b6ff21e692f-kube-api-access-f7mcn\") pod \"e2e-del-blocked-svc-kserve-85f684596-pr5rs\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:45.909897 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.909804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:45.952844 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:45.952811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:46.233625 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:46.233590    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"]
Sep 16 13:45:46.235211 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:45:46.235187    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab93e788_1429_4b26_9342_4b6ff21e692f.slice/crio-bc13a2e8375ab6505d55f4d07e7e2c24afb1bbee1fe778dd92b58c79be23ed95 WatchSource:0}: Error finding container bc13a2e8375ab6505d55f4d07e7e2c24afb1bbee1fe778dd92b58c79be23ed95: Status 404 returned error can't find the container with id bc13a2e8375ab6505d55f4d07e7e2c24afb1bbee1fe778dd92b58c79be23ed95
Sep 16 13:45:46.430867 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:46.430829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs" event={"ID":"ab93e788-1429-4b26-9342-4b6ff21e692f","Type":"ContainerStarted","Data":"bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f"}
Sep 16 13:45:46.431052 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:46.430884    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs" event={"ID":"ab93e788-1429-4b26-9342-4b6ff21e692f","Type":"ContainerStarted","Data":"bc13a2e8375ab6505d55f4d07e7e2c24afb1bbee1fe778dd92b58c79be23ed95"}
Sep 16 13:45:47.964740 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:47.964710    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"]
Sep 16 13:45:47.971780 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:47.971751    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.028060 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:48.028019    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.419585 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:48.419552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.441987 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:48.441952    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs" podUID="ab93e788-1429-4b26-9342-4b6ff21e692f" containerName="storage-initializer" containerID="cri-o://bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f" gracePeriod=60
Sep 16 13:45:48.566269 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:48.566237    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.693269 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:48.693167    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:51.187945 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.187924    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:51.259069 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259043    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab93e788-1429-4b26-9342-4b6ff21e692f-tls-certs\") pod \"ab93e788-1429-4b26-9342-4b6ff21e692f\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") "
Sep 16 13:45:51.259069 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259072    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-kserve-provision-location\") pod \"ab93e788-1429-4b26-9342-4b6ff21e692f\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") "
Sep 16 13:45:51.259283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259094    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-dshm\") pod \"ab93e788-1429-4b26-9342-4b6ff21e692f\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") "
Sep 16 13:45:51.259283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259114    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-tmp-dir\") pod \"ab93e788-1429-4b26-9342-4b6ff21e692f\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") "
Sep 16 13:45:51.259283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259196    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-home\") pod \"ab93e788-1429-4b26-9342-4b6ff21e692f\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") "
Sep 16 13:45:51.259283 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259262    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-model-cache\") pod \"ab93e788-1429-4b26-9342-4b6ff21e692f\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") "
Sep 16 13:45:51.259495 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259328    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7mcn\" (UniqueName: \"kubernetes.io/projected/ab93e788-1429-4b26-9342-4b6ff21e692f-kube-api-access-f7mcn\") pod \"ab93e788-1429-4b26-9342-4b6ff21e692f\" (UID: \"ab93e788-1429-4b26-9342-4b6ff21e692f\") "
Sep 16 13:45:51.259495 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259341    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab93e788-1429-4b26-9342-4b6ff21e692f" (UID: "ab93e788-1429-4b26-9342-4b6ff21e692f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:51.259495 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259438    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-home" (OuterVolumeSpecName: "home") pod "ab93e788-1429-4b26-9342-4b6ff21e692f" (UID: "ab93e788-1429-4b26-9342-4b6ff21e692f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:51.259626 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259516    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab93e788-1429-4b26-9342-4b6ff21e692f" (UID: "ab93e788-1429-4b26-9342-4b6ff21e692f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:51.259730 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259678    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:51.259730 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259697    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:51.259730 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.259710    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:51.261427 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.261400    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab93e788-1429-4b26-9342-4b6ff21e692f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab93e788-1429-4b26-9342-4b6ff21e692f" (UID: "ab93e788-1429-4b26-9342-4b6ff21e692f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:45:51.261427 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.261417    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-dshm" (OuterVolumeSpecName: "dshm") pod "ab93e788-1429-4b26-9342-4b6ff21e692f" (UID: "ab93e788-1429-4b26-9342-4b6ff21e692f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:51.261582 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.261490    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab93e788-1429-4b26-9342-4b6ff21e692f-kube-api-access-f7mcn" (OuterVolumeSpecName: "kube-api-access-f7mcn") pod "ab93e788-1429-4b26-9342-4b6ff21e692f" (UID: "ab93e788-1429-4b26-9342-4b6ff21e692f"). InnerVolumeSpecName "kube-api-access-f7mcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:45:51.337108 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.337044    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab93e788-1429-4b26-9342-4b6ff21e692f" (UID: "ab93e788-1429-4b26-9342-4b6ff21e692f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:45:51.360219 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.360195    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7mcn\" (UniqueName: \"kubernetes.io/projected/ab93e788-1429-4b26-9342-4b6ff21e692f-kube-api-access-f7mcn\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:51.360219 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.360215    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab93e788-1429-4b26-9342-4b6ff21e692f-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:51.360330 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.360226    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:51.360330 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.360233    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab93e788-1429-4b26-9342-4b6ff21e692f-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:45:51.459563 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.459536    2575 generic.go:356] "Generic (PLEG): container finished" podID="ab93e788-1429-4b26-9342-4b6ff21e692f" containerID="bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f" exitCode=0
Sep 16 13:45:51.459743 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.459580    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"
Sep 16 13:45:51.459743 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.459623    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs" event={"ID":"ab93e788-1429-4b26-9342-4b6ff21e692f","Type":"ContainerDied","Data":"bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f"}
Sep 16 13:45:51.459743 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.459686    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs" event={"ID":"ab93e788-1429-4b26-9342-4b6ff21e692f","Type":"ContainerDied","Data":"bc13a2e8375ab6505d55f4d07e7e2c24afb1bbee1fe778dd92b58c79be23ed95"}
Sep 16 13:45:51.459743 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.459707    2575 scope.go:117] "RemoveContainer" containerID="bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f"
Sep 16 13:45:51.469600 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.469581    2575 scope.go:117] "RemoveContainer" containerID="bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f"
Sep 16 13:45:51.469905 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:45:51.469886    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f\": container with ID starting with bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f not found: ID does not exist" containerID="bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f"
Sep 16 13:45:51.469964 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.469914    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f"} err="failed to get container status \"bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f\": rpc error: code = NotFound desc = could not find container \"bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f\": container with ID starting with bb3979600e67c672618d3e8714b25ab5f2828902f4a097987195e92250466b3f not found: ID does not exist"
Sep 16 13:45:51.501472 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.501436    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"]
Sep 16 13:45:51.507023 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:51.506999    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-85f684596-pr5rs"]
Sep 16 13:45:52.759148 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:52.759105    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:52.859699 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:52.859637    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:52.959711 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:52.959677    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:53.328553 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:53.328520    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab93e788-1429-4b26-9342-4b6ff21e692f" path="/var/lib/kubelet/pods/ab93e788-1429-4b26-9342-4b6ff21e692f/volumes"
Sep 16 13:45:53.501193 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:53.501158    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:57.159984 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:57.159944    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:57.336951 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:57.336919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:57.441199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:57.441121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:59.138182 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:59.138149    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:59.211993 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:45:59.211963    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:03.206424 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:03.206387    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:03.307786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:03.307752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:06.410970 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:06.410936    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:06.508442 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:06.508411    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:12.893530 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:12.893498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:13.889005 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:13.888965    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:17.570135 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.570103    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"]
Sep 16 13:46:17.570505 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.570491    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab93e788-1429-4b26-9342-4b6ff21e692f" containerName="storage-initializer"
Sep 16 13:46:17.570554 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.570506    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab93e788-1429-4b26-9342-4b6ff21e692f" containerName="storage-initializer"
Sep 16 13:46:17.570604 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.570567    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab93e788-1429-4b26-9342-4b6ff21e692f" containerName="storage-initializer"
Sep 16 13:46:17.578863 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.578836    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:17.583401 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.583376    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 16 13:46:17.584815 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.584797    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 16 13:46:17.585335 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.585318    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 16 13:46:17.586160 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.586141    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-5v95b\""
Sep 16 13:46:17.590457 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.590439    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"]
Sep 16 13:46:17.596363 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.596342    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:17.678831 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.678803    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvkt\" (UniqueName: \"kubernetes.io/projected/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kube-api-access-brvkt\") pod \"e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79\" (UID: \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:17.678989 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.678891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79\" (UID: \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:17.779701 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.779673    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvkt\" (UniqueName: \"kubernetes.io/projected/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kube-api-access-brvkt\") pod \"e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79\" (UID: \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:17.779873 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.779766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79\" (UID: \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:17.780136 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.780113    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79\" (UID: \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:17.793972 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.793940    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvkt\" (UniqueName: \"kubernetes.io/projected/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kube-api-access-brvkt\") pod \"e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79\" (UID: \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:17.889495 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:17.889470    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:18.165191 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:18.165166    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"]
Sep 16 13:46:18.166943 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:46:18.166911    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cb1bb8_3418_4ed7_ab3c_511aaed4e58f.slice/crio-4dd0933eb208e9640e350156c942684133c73d8e5c7f99bd3d78e0294f845ee4 WatchSource:0}: Error finding container 4dd0933eb208e9640e350156c942684133c73d8e5c7f99bd3d78e0294f845ee4: Status 404 returned error can't find the container with id 4dd0933eb208e9640e350156c942684133c73d8e5c7f99bd3d78e0294f845ee4
Sep 16 13:46:18.601439 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:18.601355    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79" event={"ID":"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f","Type":"ContainerStarted","Data":"d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931"}
Sep 16 13:46:18.601439 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:18.601416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79" event={"ID":"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f","Type":"ContainerStarted","Data":"4dd0933eb208e9640e350156c942684133c73d8e5c7f99bd3d78e0294f845ee4"}
Sep 16 13:46:18.984389 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:18.984355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:19.555053 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:19.554984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:19.656542 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:19.656515    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:22.624062 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:22.624032    2575 generic.go:356] "Generic (PLEG): container finished" podID="e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" containerID="d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931" exitCode=0
Sep 16 13:46:22.624368 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:22.624110    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79" event={"ID":"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f","Type":"ContainerDied","Data":"d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931"}
Sep 16 13:46:23.638833 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:23.638795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79" event={"ID":"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f","Type":"ContainerStarted","Data":"386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f"}
Sep 16 13:46:23.660196 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:23.660147    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79" podStartSLOduration=2.334193931 podStartE2EDuration="6.66013151s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.325937579s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:23.657254012 +0000 UTC m=+3504.838396687" watchObservedRunningTime="2026-09-16 13:46:23.66013151 +0000 UTC m=+3504.841274162"
Sep 16 13:46:27.907736 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.907703    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"]
Sep 16 13:46:27.912737 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.912714    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:27.915759 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.915737    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 16 13:46:27.915875 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.915739    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 13:46:27.915875 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.915848    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 13:46:27.916103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.916082    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 13:46:27.916103 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.916092    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:46:27.916983 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.916953    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-6xmtf\""
Sep 16 13:46:27.923444 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.923424    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"]
Sep 16 13:46:27.956771 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.956736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-model-cache\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:27.956771 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.956782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p29d\" (UniqueName: \"kubernetes.io/projected/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kube-api-access-5p29d\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:27.957026 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.956867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-cabundle-cert\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:27.957026 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.956898    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:27.957026 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.956939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-dshm\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:27.957026 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.956971    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-home\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:27.957248 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.957057    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tmp-dir\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:27.957248 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:27.957112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tls-certs\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.027312 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.027284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:28.058383 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058351    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-dshm\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.058511 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-home\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.058511 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tmp-dir\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.058511 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tls-certs\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.058674 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-model-cache\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.058674 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p29d\" (UniqueName: \"kubernetes.io/projected/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kube-api-access-5p29d\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.058674 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-cabundle-cert\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.058839 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058699    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.058890 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058866    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-home\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.058949 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.058910    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tmp-dir\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.059172 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.059146    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-model-cache\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.059422 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.059402    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.059734 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.059711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-cabundle-cert\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.060979 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.060952    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-dshm\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.061092 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.061069    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tls-certs\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.068160 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.068136    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p29d\" (UniqueName: \"kubernetes.io/projected/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kube-api-access-5p29d\") pod \"custom-route-timeout-test-kserve-74dcbd8565-zgzcj\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.224236 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.224158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:28.367047 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.367010    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:28.515762 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.515664    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"]
Sep 16 13:46:28.518845 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:46:28.518816    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5986a3c7_ab3d_47a1_ae88_24e77a4a6a79.slice/crio-52940387e4e268bb12820973484ff0cb12f9fc0b51f8c61be57f452438356d2a WatchSource:0}: Error finding container 52940387e4e268bb12820973484ff0cb12f9fc0b51f8c61be57f452438356d2a: Status 404 returned error can't find the container with id 52940387e4e268bb12820973484ff0cb12f9fc0b51f8c61be57f452438356d2a
Sep 16 13:46:28.678217 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.678176    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" event={"ID":"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79","Type":"ContainerStarted","Data":"9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09"}
Sep 16 13:46:28.678434 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:28.678261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" event={"ID":"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79","Type":"ContainerStarted","Data":"52940387e4e268bb12820973484ff0cb12f9fc0b51f8c61be57f452438356d2a"}
Sep 16 13:46:30.903741 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:30.903704    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"]
Sep 16 13:46:30.904220 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:30.903926    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79" podUID="e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" containerName="main" containerID="cri-o://386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f" gracePeriod=30
Sep 16 13:46:30.920484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:30.920449    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:31.059962 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:31.059924    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:31.697056 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:31.697017    2575 generic.go:356] "Generic (PLEG): container finished" podID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerID="9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09" exitCode=0
Sep 16 13:46:31.697244 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:31.697068    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" event={"ID":"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79","Type":"ContainerDied","Data":"9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09"}
Sep 16 13:46:32.014805 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:32.014715    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:32.109806 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:32.109770    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:32.706737 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:32.706696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" event={"ID":"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79","Type":"ContainerStarted","Data":"aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa"}
Sep 16 13:46:32.733700 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:32.733617    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podStartSLOduration=3.240088031 podStartE2EDuration="5.733593212s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.493505181s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:32.731115325 +0000 UTC m=+3513.912257982" watchObservedRunningTime="2026-09-16 13:46:32.733593212 +0000 UTC m=+3513.914735873"
Sep 16 13:46:33.224839 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:33.224797    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:37.096531 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:37.096500    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:38.224833 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:38.224792    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:46:38.226878 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:38.226844    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:46:41.759608 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.759586    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:41.774637 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.774598    2575 generic.go:356] "Generic (PLEG): container finished" podID="e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" containerID="386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f" exitCode=1
Sep 16 13:46:41.774786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.774671    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"
Sep 16 13:46:41.774786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.774692    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79" event={"ID":"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f","Type":"ContainerDied","Data":"386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f"}
Sep 16 13:46:41.774786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.774753    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79" event={"ID":"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f","Type":"ContainerDied","Data":"4dd0933eb208e9640e350156c942684133c73d8e5c7f99bd3d78e0294f845ee4"}
Sep 16 13:46:41.774786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.774779    2575 scope.go:117] "RemoveContainer" containerID="386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f"
Sep 16 13:46:41.785709 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.785690    2575 scope.go:117] "RemoveContainer" containerID="d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931"
Sep 16 13:46:41.797856 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.797835    2575 scope.go:117] "RemoveContainer" containerID="386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f"
Sep 16 13:46:41.798195 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:46:41.798170    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f\": container with ID starting with 386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f not found: ID does not exist" containerID="386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f"
Sep 16 13:46:41.798264 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.798207    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f"} err="failed to get container status \"386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f\": rpc error: code = NotFound desc = could not find container \"386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f\": container with ID starting with 386c049fc8f29c9cabb62b7f73a83e703195d18ef5aa9558f1d9d83bf1518a9f not found: ID does not exist"
Sep 16 13:46:41.798264 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.798233    2575 scope.go:117] "RemoveContainer" containerID="d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931"
Sep 16 13:46:41.798548 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:46:41.798526    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931\": container with ID starting with d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931 not found: ID does not exist" containerID="d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931"
Sep 16 13:46:41.798611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.798557    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931"} err="failed to get container status \"d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931\": rpc error: code = NotFound desc = could not find container \"d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931\": container with ID starting with d2e6dc45dfa4a9c29c4c903382a071bc142e197cf4febeaaa12b22bd2d359931 not found: ID does not exist"
Sep 16 13:46:41.885027 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.884998    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kserve-provision-location\") pod \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\" (UID: \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\") "
Sep 16 13:46:41.885223 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.885060    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brvkt\" (UniqueName: \"kubernetes.io/projected/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kube-api-access-brvkt\") pod \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\" (UID: \"e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f\") "
Sep 16 13:46:41.887399 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.887374    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kube-api-access-brvkt" (OuterVolumeSpecName: "kube-api-access-brvkt") pod "e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" (UID: "e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f"). InnerVolumeSpecName "kube-api-access-brvkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:46:41.948988 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.948947    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" (UID: "e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:46:41.986510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.986477    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:46:41.986510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:41.986504    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvkt\" (UniqueName: \"kubernetes.io/projected/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f-kube-api-access-brvkt\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:46:42.108401 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:42.108368    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"]
Sep 16 13:46:42.110297 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:42.110270    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-9cbccbf64-xsb79"]
Sep 16 13:46:43.329464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:43.329420    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" path="/var/lib/kubelet/pods/e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f/volumes"
Sep 16 13:46:43.869186 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:43.869152    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:43.943153 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:43.943112    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:46.809755 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.809729    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"]
Sep 16 13:46:46.810149 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.810077    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" containerName="storage-initializer"
Sep 16 13:46:46.810149 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.810087    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" containerName="storage-initializer"
Sep 16 13:46:46.810149 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.810096    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" containerName="main"
Sep 16 13:46:46.810149 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.810102    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" containerName="main"
Sep 16 13:46:46.810301 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.810181    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cb1bb8-3418-4ed7-ab3c-511aaed4e58f" containerName="main"
Sep 16 13:46:46.813366 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.813350    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:46.816819 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.816791    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:46:46.816819 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.816790    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 13:46:46.817035 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.816869    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-f85r9\""
Sep 16 13:46:46.817133 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.817108    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 13:46:46.817418 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.817404    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 13:46:46.817782 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.817766    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 13:46:46.831070 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.831049    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"]
Sep 16 13:46:46.908694 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.908630    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:46.928296 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.928263    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kserve-provision-location\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:46.928468 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.928305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tmp-dir\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:46.928468 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.928348    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb074878-9c5d-41b7-a61f-a5ccf02287dc-cabundle-cert\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:46.928468 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.928389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-model-cache\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:46.928468 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.928420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-home\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:46.928468 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.928452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tls-certs\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:46.928694 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.928487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvktv\" (UniqueName: \"kubernetes.io/projected/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kube-api-access-fvktv\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:46.928694 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:46.928575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-dshm\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.029704 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.029638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-dshm\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.029704 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.029704    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kserve-provision-location\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.029991 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.029736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tmp-dir\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.029991 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.029861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb074878-9c5d-41b7-a61f-a5ccf02287dc-cabundle-cert\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.029991 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.029923    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-model-cache\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.029991 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.029966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-home\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.030260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.030003    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tls-certs\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.030260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.030048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvktv\" (UniqueName: \"kubernetes.io/projected/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kube-api-access-fvktv\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.030260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.030121    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tmp-dir\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.030260 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.030169    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kserve-provision-location\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.030461 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.030357    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-model-cache\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.030598 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.030578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb074878-9c5d-41b7-a61f-a5ccf02287dc-cabundle-cert\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.030795 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.030765    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-home\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.032221 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.032202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-dshm\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.032825 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.032806    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tls-certs\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.043793 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.043770    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvktv\" (UniqueName: \"kubernetes.io/projected/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kube-api-access-fvktv\") pod \"stop-feature-test-kserve-68fc8f9464-268vk\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.127625 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.127591    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:47.144303 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.144178    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:47.416860 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.416826    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"]
Sep 16 13:46:47.417687 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:46:47.417658    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb074878_9c5d_41b7_a61f_a5ccf02287dc.slice/crio-7006e83afb0039706b62c008229e12706753dff97cfeab9d7b510b2423783cf5 WatchSource:0}: Error finding container 7006e83afb0039706b62c008229e12706753dff97cfeab9d7b510b2423783cf5: Status 404 returned error can't find the container with id 7006e83afb0039706b62c008229e12706753dff97cfeab9d7b510b2423783cf5
Sep 16 13:46:47.813023 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.812913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" event={"ID":"bb074878-9c5d-41b7-a61f-a5ccf02287dc","Type":"ContainerStarted","Data":"12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f"}
Sep 16 13:46:47.813023 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:47.812978    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" event={"ID":"bb074878-9c5d-41b7-a61f-a5ccf02287dc","Type":"ContainerStarted","Data":"7006e83afb0039706b62c008229e12706753dff97cfeab9d7b510b2423783cf5"}
Sep 16 13:46:48.225351 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:48.225308    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:46:50.839090 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:50.839051    2575 generic.go:356] "Generic (PLEG): container finished" podID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerID="12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f" exitCode=0
Sep 16 13:46:50.839449 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:50.839105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" event={"ID":"bb074878-9c5d-41b7-a61f-a5ccf02287dc","Type":"ContainerDied","Data":"12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f"}
Sep 16 13:46:51.848665 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:51.848617    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" event={"ID":"bb074878-9c5d-41b7-a61f-a5ccf02287dc","Type":"ContainerStarted","Data":"d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9"}
Sep 16 13:46:51.873911 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:51.873844    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podStartSLOduration=3.448987402 podStartE2EDuration="5.87382375s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.424836348s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:51.873053656 +0000 UTC m=+3533.054196310" watchObservedRunningTime="2026-09-16 13:46:51.87382375 +0000 UTC m=+3533.054966405"
Sep 16 13:46:52.127881 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:52.127849    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:57.128452 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:57.128409    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:46:57.130343 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:57.130306    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:46:58.225243 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:46:58.225143    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:47:01.449077 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:01.449026    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:02.730959 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:02.730917    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:07.128674 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:07.128609    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:47:08.225436 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:08.225396    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:47:17.128382 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:17.128334    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:47:18.225607 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:18.225567    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:47:27.128757 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:27.128700    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:47:28.224973 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:28.224934    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:47:37.128541 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:37.128490    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:47:38.225527 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:38.225475    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:47:45.129595 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:45.129561    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:47.129032 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:47.128991    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:47:48.224868 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:48.224827    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:47:57.128742 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:57.128619    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:47:58.225638 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:47:58.225532    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:48:07.128618 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:07.128574    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:48:08.224806 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:08.224754    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:48:17.128985 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:17.128937    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:48:18.225008 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:18.224967    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:48:27.128502 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:27.128457    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:48:28.224820 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:28.224773    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:48:37.128920 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:37.128865    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:48:38.225693 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:38.225626    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:48:47.128337 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:47.128291    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:48:48.235409 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:48.235370    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:48:48.243348 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:48.243322    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:48:55.830784 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:55.830748    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"]
Sep 16 13:48:55.831564 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:55.831534    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:55.923845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:55.923809    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:57.128566 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:48:57.128522    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:49:07.138177 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:07.138146    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:49:07.146091 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:07.146069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:49:08.893571 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:08.893539    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"]
Sep 16 13:49:08.926394 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:08.926356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:08.995072 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:08.995045    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:09.122142 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:09.122097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:10.854988 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:10.854950    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" containerID="cri-o://aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa" gracePeriod=45
Sep 16 13:49:11.236168 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.236118    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 13:49:11.603761 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.603738    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:49:11.685178 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.685143    2575 generic.go:356] "Generic (PLEG): container finished" podID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerID="aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa" exitCode=0
Sep 16 13:49:11.685331 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.685191    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" event={"ID":"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79","Type":"ContainerDied","Data":"aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa"}
Sep 16 13:49:11.685331 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.685245    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj" event={"ID":"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79","Type":"ContainerDied","Data":"52940387e4e268bb12820973484ff0cb12f9fc0b51f8c61be57f452438356d2a"}
Sep 16 13:49:11.685331 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.685261    2575 scope.go:117] "RemoveContainer" containerID="aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa"
Sep 16 13:49:11.685331 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.685197    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"
Sep 16 13:49:11.696484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.696466    2575 scope.go:117] "RemoveContainer" containerID="9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09"
Sep 16 13:49:11.704752 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.704731    2575 scope.go:117] "RemoveContainer" containerID="aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa"
Sep 16 13:49:11.705014 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:49:11.704994    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa\": container with ID starting with aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa not found: ID does not exist" containerID="aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa"
Sep 16 13:49:11.705076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.705023    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa"} err="failed to get container status \"aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa\": rpc error: code = NotFound desc = could not find container \"aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa\": container with ID starting with aa4575d2a0f362b03d404cb6b5db8996a9877b71fbf1cad3f5b224827c82d2fa not found: ID does not exist"
Sep 16 13:49:11.705076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.705042    2575 scope.go:117] "RemoveContainer" containerID="9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09"
Sep 16 13:49:11.705299 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:49:11.705281    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09\": container with ID starting with 9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09 not found: ID does not exist" containerID="9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09"
Sep 16 13:49:11.705341 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.705306    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09"} err="failed to get container status \"9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09\": rpc error: code = NotFound desc = could not find container \"9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09\": container with ID starting with 9502a568066107df90b21b9c6593aeb8f0b28354c0b84d03a71fe7fa4ea60a09 not found: ID does not exist"
Sep 16 13:49:11.723570 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.723550    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p29d\" (UniqueName: \"kubernetes.io/projected/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kube-api-access-5p29d\") pod \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") "
Sep 16 13:49:11.723666 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.723595    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tls-certs\") pod \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") "
Sep 16 13:49:11.723666 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.723631    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kserve-provision-location\") pod \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") "
Sep 16 13:49:11.723782 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.723675    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-home\") pod \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") "
Sep 16 13:49:11.723782 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.723744    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tmp-dir\") pod \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") "
Sep 16 13:49:11.723888 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.723779    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-dshm\") pod \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") "
Sep 16 13:49:11.723888 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.723814    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-cabundle-cert\") pod \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") "
Sep 16 13:49:11.723888 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.723835    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-model-cache\") pod \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\" (UID: \"5986a3c7-ab3d-47a1-ae88-24e77a4a6a79\") "
Sep 16 13:49:11.724639 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.724330    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" (UID: "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:49:11.724639 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.724358    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-model-cache" (OuterVolumeSpecName: "model-cache") pod "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" (UID: "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:11.724639 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.724531    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-home" (OuterVolumeSpecName: "home") pod "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" (UID: "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:11.726179 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.726159    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" (UID: "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:49:11.726319 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.726296    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kube-api-access-5p29d" (OuterVolumeSpecName: "kube-api-access-5p29d") pod "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" (UID: "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79"). InnerVolumeSpecName "kube-api-access-5p29d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:49:11.726561 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.726543    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-dshm" (OuterVolumeSpecName: "dshm") pod "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" (UID: "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:11.742163 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.742134    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" (UID: "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:11.751002 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.750983    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" (UID: "5986a3c7-ab3d-47a1-ae88-24e77a4a6a79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:11.824906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.824856    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:11.824906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.824877    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:11.824906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.824885    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:11.824906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.824894    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:11.824906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.824903    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p29d\" (UniqueName: \"kubernetes.io/projected/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kube-api-access-5p29d\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:11.825105 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.824911    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:11.825105 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.824920    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:11.825105 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:11.824927    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:12.010630 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:12.010597    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"]
Sep 16 13:49:12.016175 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:12.016151    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-74dcbd8565-zgzcj"]
Sep 16 13:49:13.329180 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:13.329144    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" path="/var/lib/kubelet/pods/5986a3c7-ab3d-47a1-ae88-24e77a4a6a79/volumes"
Sep 16 13:49:22.090606 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.090530    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"]
Sep 16 13:49:22.090988 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.090972    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main"
Sep 16 13:49:22.091033 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.090990    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main"
Sep 16 13:49:22.091033 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.091000    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="storage-initializer"
Sep 16 13:49:22.091033 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.091006    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="storage-initializer"
Sep 16 13:49:22.091126 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.091080    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5986a3c7-ab3d-47a1-ae88-24e77a4a6a79" containerName="main"
Sep 16 13:49:22.094100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.094079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.105508 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.105485    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"]
Sep 16 13:49:22.108851 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.108828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tmp-dir\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.108963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.108866    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-cabundle-cert\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.108963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.108902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-model-cache\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.108963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.108923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tls-certs\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.108963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.108952    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-home\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.109164 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.109053    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kserve-provision-location\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.109164 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.109089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwr5g\" (UniqueName: \"kubernetes.io/projected/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kube-api-access-jwr5g\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.109164 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.109119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-dshm\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.110125 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.110101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:22.209862 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.209828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-cabundle-cert\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.209877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-model-cache\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210015 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tls-certs\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210131 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210063    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-home\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210131 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kserve-provision-location\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210144    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwr5g\" (UniqueName: \"kubernetes.io/projected/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kube-api-access-jwr5g\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210182    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-dshm\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tmp-dir\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210402 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-model-cache\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210402 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210354    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-home\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210596 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210572    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kserve-provision-location\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210754 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210628    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-cabundle-cert\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.210754 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.210590    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tmp-dir\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.212502 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.212478    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-dshm\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.213098 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.213079    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tls-certs\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.218732 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.218635    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwr5g\" (UniqueName: \"kubernetes.io/projected/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kube-api-access-jwr5g\") pod \"stop-feature-test-kserve-68fc8f9464-m682b\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.296264 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.296227    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:22.404809 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.404779    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:22.683389 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.683231    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"]
Sep 16 13:49:22.686184 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:49:22.686154    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad0e1c8_c7bf_4abf_a5a1_4e1d4376a70c.slice/crio-65f32ff3a9c83a2d113d89a37c3d6ca585ecb6396b2fa7e1e9849e1e0c5450d7 WatchSource:0}: Error finding container 65f32ff3a9c83a2d113d89a37c3d6ca585ecb6396b2fa7e1e9849e1e0c5450d7: Status 404 returned error can't find the container with id 65f32ff3a9c83a2d113d89a37c3d6ca585ecb6396b2fa7e1e9849e1e0c5450d7
Sep 16 13:49:22.740390 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:22.740363    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" event={"ID":"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c","Type":"ContainerStarted","Data":"65f32ff3a9c83a2d113d89a37c3d6ca585ecb6396b2fa7e1e9849e1e0c5450d7"}
Sep 16 13:49:23.750371 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:23.750330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" event={"ID":"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c","Type":"ContainerStarted","Data":"d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f"}
Sep 16 13:49:23.918344 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:23.918308    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" containerID="cri-o://d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9" gracePeriod=45
Sep 16 13:49:24.138838 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.138785    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 13:49:24.687749 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.687715    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:49:24.737276 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737192    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-home\") pod \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") "
Sep 16 13:49:24.737276 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737266    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvktv\" (UniqueName: \"kubernetes.io/projected/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kube-api-access-fvktv\") pod \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") "
Sep 16 13:49:24.737691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737299    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tmp-dir\") pod \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") "
Sep 16 13:49:24.737691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737365    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-model-cache\") pod \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") "
Sep 16 13:49:24.737691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737390    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-dshm\") pod \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") "
Sep 16 13:49:24.737691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737430    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tls-certs\") pod \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") "
Sep 16 13:49:24.737691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737465    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kserve-provision-location\") pod \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") "
Sep 16 13:49:24.737691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737513    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb074878-9c5d-41b7-a61f-a5ccf02287dc-cabundle-cert\") pod \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\" (UID: \"bb074878-9c5d-41b7-a61f-a5ccf02287dc\") "
Sep 16 13:49:24.738012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737699    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb074878-9c5d-41b7-a61f-a5ccf02287dc" (UID: "bb074878-9c5d-41b7-a61f-a5ccf02287dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:24.738012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737801    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:24.738012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.737985    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb074878-9c5d-41b7-a61f-a5ccf02287dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bb074878-9c5d-41b7-a61f-a5ccf02287dc" (UID: "bb074878-9c5d-41b7-a61f-a5ccf02287dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:49:24.738182 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.738056    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-home" (OuterVolumeSpecName: "home") pod "bb074878-9c5d-41b7-a61f-a5ccf02287dc" (UID: "bb074878-9c5d-41b7-a61f-a5ccf02287dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:24.741004 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.740981    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-dshm" (OuterVolumeSpecName: "dshm") pod "bb074878-9c5d-41b7-a61f-a5ccf02287dc" (UID: "bb074878-9c5d-41b7-a61f-a5ccf02287dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:24.742433 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.742371    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kube-api-access-fvktv" (OuterVolumeSpecName: "kube-api-access-fvktv") pod "bb074878-9c5d-41b7-a61f-a5ccf02287dc" (UID: "bb074878-9c5d-41b7-a61f-a5ccf02287dc"). InnerVolumeSpecName "kube-api-access-fvktv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:49:24.743227 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.743022    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb074878-9c5d-41b7-a61f-a5ccf02287dc" (UID: "bb074878-9c5d-41b7-a61f-a5ccf02287dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:49:24.757707 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.757352    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb074878-9c5d-41b7-a61f-a5ccf02287dc" (UID: "bb074878-9c5d-41b7-a61f-a5ccf02287dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:24.762972 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.762937    2575 generic.go:356] "Generic (PLEG): container finished" podID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerID="d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9" exitCode=0
Sep 16 13:49:24.763773 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.763681    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"
Sep 16 13:49:24.763773 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.763711    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" event={"ID":"bb074878-9c5d-41b7-a61f-a5ccf02287dc","Type":"ContainerDied","Data":"d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9"}
Sep 16 13:49:24.763956 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.763777    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk" event={"ID":"bb074878-9c5d-41b7-a61f-a5ccf02287dc","Type":"ContainerDied","Data":"7006e83afb0039706b62c008229e12706753dff97cfeab9d7b510b2423783cf5"}
Sep 16 13:49:24.763956 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.763808    2575 scope.go:117] "RemoveContainer" containerID="d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9"
Sep 16 13:49:24.771604 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.771576    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb074878-9c5d-41b7-a61f-a5ccf02287dc" (UID: "bb074878-9c5d-41b7-a61f-a5ccf02287dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:49:24.776554 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.776525    2575 scope.go:117] "RemoveContainer" containerID="12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f"
Sep 16 13:49:24.790522 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.790361    2575 scope.go:117] "RemoveContainer" containerID="d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9"
Sep 16 13:49:24.791162 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:49:24.791096    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9\": container with ID starting with d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9 not found: ID does not exist" containerID="d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9"
Sep 16 13:49:24.791162 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.791145    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9"} err="failed to get container status \"d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9\": rpc error: code = NotFound desc = could not find container \"d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9\": container with ID starting with d6172372a7a8712103c2ab2669033da1c3f56963bc2dc507d92abf08596389a9 not found: ID does not exist"
Sep 16 13:49:24.791400 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.791171    2575 scope.go:117] "RemoveContainer" containerID="12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f"
Sep 16 13:49:24.792027 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:49:24.791996    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f\": container with ID starting with 12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f not found: ID does not exist" containerID="12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f"
Sep 16 13:49:24.792178 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.792035    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f"} err="failed to get container status \"12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f\": rpc error: code = NotFound desc = could not find container \"12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f\": container with ID starting with 12cfc1b8d69b8f304a6b53d62284a320623d7b2ccaf242a57f4ed332ed97c42f not found: ID does not exist"
Sep 16 13:49:24.838834 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.838797    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb074878-9c5d-41b7-a61f-a5ccf02287dc-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:24.838834 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.838827    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:24.839017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.838840    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvktv\" (UniqueName: \"kubernetes.io/projected/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kube-api-access-fvktv\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:24.839017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.838854    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:24.839017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.838868    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:24.839017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.838879    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb074878-9c5d-41b7-a61f-a5ccf02287dc-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:24.839017 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:24.838891    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb074878-9c5d-41b7-a61f-a5ccf02287dc-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:49:25.113570 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:25.113544    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"]
Sep 16 13:49:25.117660 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:25.117614    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-268vk"]
Sep 16 13:49:25.329692 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:25.329604    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" path="/var/lib/kubelet/pods/bb074878-9c5d-41b7-a61f-a5ccf02287dc/volumes"
Sep 16 13:49:25.769556 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:25.769527    2575 generic.go:356] "Generic (PLEG): container finished" podID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerID="d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f" exitCode=0
Sep 16 13:49:25.769946 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:25.769601    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" event={"ID":"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c","Type":"ContainerDied","Data":"d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f"}
Sep 16 13:49:26.776724 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:26.776680    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" event={"ID":"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c","Type":"ContainerStarted","Data":"82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151"}
Sep 16 13:49:26.811840 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:26.811784    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podStartSLOduration=2.528940756 podStartE2EDuration="4.81176802s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.282827264s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:26.808186706 +0000 UTC m=+3687.989329361" watchObservedRunningTime="2026-09-16 13:49:26.81176802 +0000 UTC m=+3687.992910674"
Sep 16 13:49:27.405119 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:27.405065    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:32.405115 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:32.405077    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:49:32.406572 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:32.406547    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:49:32.830126 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:32.830016    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:33.464899 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:33.464851    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:33.531010 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:33.530948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:42.405994 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:42.405952    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:49:45.957467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:45.957427    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:46.054963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.054892    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:46.168995 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.168957    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"]
Sep 16 13:49:46.169358 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.169343    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="storage-initializer"
Sep 16 13:49:46.169405 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.169362    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="storage-initializer"
Sep 16 13:49:46.169405 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.169377    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main"
Sep 16 13:49:46.169405 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.169385    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main"
Sep 16 13:49:46.169503 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.169472    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb074878-9c5d-41b7-a61f-a5ccf02287dc" containerName="main"
Sep 16 13:49:46.172290 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.172268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.175061 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.175034    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 16 13:49:46.175237 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.175212    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 13:49:46.175366 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.175327    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 13:49:46.176265 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.176248    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-6lf6g\""
Sep 16 13:49:46.185954 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.185928    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"]
Sep 16 13:49:46.201599 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.201562    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:46.303625 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.303547    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:46.319456 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.319422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.319585 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.319470    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.319585 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.319527    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrld2\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-kube-api-access-lrld2\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.319585 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.319559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.319754 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.319597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.319754 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.319629    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.319754 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.319689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.319903 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.319757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.319903 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.319805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421028 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.420994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421028 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421282 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrld2\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-kube-api-access-lrld2\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421282 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421282 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421282 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421282 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421159    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421562 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421491    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421562 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421501    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421562 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421512    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421562 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421814 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421566    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.421867 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.421813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.422133 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.422115    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.423526 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.423507    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.424204 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.424183    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.430350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.430321    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.430438 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.430419    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrld2\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-kube-api-access-lrld2\") pod \"router-gateway-1-openshift-default-56bd58cf4d-zn2jn\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.485026 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.484989    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:46.785066 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.785026    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"]
Sep 16 13:49:46.787736 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:49:46.787698    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa5c41c_6f0a_40e3_8838_f060df5ddc16.slice/crio-18ad0505e449f8e9d1587825d7165672b72779cb41fdccac69d41ed86a37b190 WatchSource:0}: Error finding container 18ad0505e449f8e9d1587825d7165672b72779cb41fdccac69d41ed86a37b190: Status 404 returned error can't find the container with id 18ad0505e449f8e9d1587825d7165672b72779cb41fdccac69d41ed86a37b190
Sep 16 13:49:46.790078 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.790040    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:49:46.790164 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.790124    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:49:46.790208 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.790166    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:49:46.886139 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.886099    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" event={"ID":"1aa5c41c-6f0a-40e3-8838-f060df5ddc16","Type":"ContainerStarted","Data":"3f46dcb25e51fe45a9fddb5427a8c9b6f158a027a5febb3de3ad4e9fa4f41ba6"}
Sep 16 13:49:46.886308 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.886170    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" event={"ID":"1aa5c41c-6f0a-40e3-8838-f060df5ddc16","Type":"ContainerStarted","Data":"18ad0505e449f8e9d1587825d7165672b72779cb41fdccac69d41ed86a37b190"}
Sep 16 13:49:46.911412 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:46.911354    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" podStartSLOduration=0.91133318 podStartE2EDuration="911.33318ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:49:46.909830543 +0000 UTC m=+3708.090973221" watchObservedRunningTime="2026-09-16 13:49:46.91133318 +0000 UTC m=+3708.092475834"
Sep 16 13:49:47.485508 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:47.485477    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:47.490291 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:47.490266    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:47.892808 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:47.892770    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:47.893770 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:47.893749    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:49.763716 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:49.763679    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:52.405337 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:52.405293    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:49:52.898419 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:52.898366    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:57.953282 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:57.953191    2575 generic.go:356] "Generic (PLEG): container finished" podID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerID="3f46dcb25e51fe45a9fddb5427a8c9b6f158a027a5febb3de3ad4e9fa4f41ba6" exitCode=0
Sep 16 13:49:57.953282 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:57.953263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" event={"ID":"1aa5c41c-6f0a-40e3-8838-f060df5ddc16","Type":"ContainerDied","Data":"3f46dcb25e51fe45a9fddb5427a8c9b6f158a027a5febb3de3ad4e9fa4f41ba6"}
Sep 16 13:49:57.953861 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:57.953816    2575 scope.go:117] "RemoveContainer" containerID="3f46dcb25e51fe45a9fddb5427a8c9b6f158a027a5febb3de3ad4e9fa4f41ba6"
Sep 16 13:49:57.954856 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:57.954819    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:49:57.954967 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:57.954923    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:49:57.955034 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:57.954964    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:49:58.694387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:58.694355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:58.795375 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:58.795331    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:58.960932 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:58.960829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" event={"ID":"1aa5c41c-6f0a-40e3-8838-f060df5ddc16","Type":"ContainerStarted","Data":"012de13200806f8809b4bb70caed65aafa2b6249adf8a860980a0e91572856a5"}
Sep 16 13:49:59.485918 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:59.485883    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:49:59.487197 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:49:59.487160    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 13:50:00.486067 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:00.486025    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 13:50:01.485946 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:01.485906    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 13:50:02.405353 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:02.405307    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:50:02.485945 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:02.485901    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 13:50:02.893224 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:02.893187    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:50:03.485933 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:03.485890    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 13:50:04.489691 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:04.489636    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:50:04.490567 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:04.490549    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:50:11.476464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.476429    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"]
Sep 16 13:50:11.480018 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.479996    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.483142 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.483118    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:50:11.483273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.483182    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-ghtcn\""
Sep 16 13:50:11.483347 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.483119    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 16 13:50:11.483411 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.483118    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 13:50:11.493578 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.493557    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"]
Sep 16 13:50:11.509310 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.509255    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:11.555974 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.555944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-cabundle-cert\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.556097 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.556009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tmp-dir\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.556097 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.556059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kserve-provision-location\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.556197 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.556105    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-home\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.556197 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.556139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-dshm\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.556197 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.556163    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-model-cache\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.556321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.556226    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7t7v\" (UniqueName: \"kubernetes.io/projected/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kube-api-access-t7t7v\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.556321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.556270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tls-certs\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.657575 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.657544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-cabundle-cert\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.657733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.657609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tmp-dir\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.657733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.657637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kserve-provision-location\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.657733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.657689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-home\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.657733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.657720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-dshm\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.657963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.657743    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-model-cache\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.657963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.657780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7t7v\" (UniqueName: \"kubernetes.io/projected/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kube-api-access-t7t7v\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.657963 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.657809    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tls-certs\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.658128 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.658030    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kserve-provision-location\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.658128 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.658062    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tmp-dir\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.658254 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.658137    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-home\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.658310 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.658280    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-model-cache\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.658427 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.658406    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-cabundle-cert\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.660187 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.660161    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-dshm\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.660407 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.660390    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tls-certs\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.666336 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.666313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7t7v\" (UniqueName: \"kubernetes.io/projected/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kube-api-access-t7t7v\") pod \"router-with-refs-test-kserve-77f8478c8f-bptwz\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:11.793394 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:11.793310    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:12.074495 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:12.074470    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"]
Sep 16 13:50:12.076218 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:50:12.076184    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2c6dcf_88cc_4991_8683_63f590a7d3e9.slice/crio-9996a1bfe5241cc8aa343fee1f62e9bf71ccc550e55aae38a909cf69d5322486 WatchSource:0}: Error finding container 9996a1bfe5241cc8aa343fee1f62e9bf71ccc550e55aae38a909cf69d5322486: Status 404 returned error can't find the container with id 9996a1bfe5241cc8aa343fee1f62e9bf71ccc550e55aae38a909cf69d5322486
Sep 16 13:50:12.076422 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:12.076396    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:12.186123 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:12.186065    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:12.406262 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:12.406213    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:50:13.047195 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:13.047150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" event={"ID":"6c2c6dcf-88cc-4991-8683-63f590a7d3e9","Type":"ContainerStarted","Data":"3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c"}
Sep 16 13:50:13.047706 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:13.047223    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" event={"ID":"6c2c6dcf-88cc-4991-8683-63f590a7d3e9","Type":"ContainerStarted","Data":"9996a1bfe5241cc8aa343fee1f62e9bf71ccc550e55aae38a909cf69d5322486"}
Sep 16 13:50:15.062538 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:15.062503    2575 generic.go:356] "Generic (PLEG): container finished" podID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerID="3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c" exitCode=0
Sep 16 13:50:15.062927 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:15.062585    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" event={"ID":"6c2c6dcf-88cc-4991-8683-63f590a7d3e9","Type":"ContainerDied","Data":"3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c"}
Sep 16 13:50:16.070695 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:16.070629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" event={"ID":"6c2c6dcf-88cc-4991-8683-63f590a7d3e9","Type":"ContainerStarted","Data":"b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19"}
Sep 16 13:50:16.095752 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:16.095698    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podStartSLOduration=2.697228805 podStartE2EDuration="5.095678537s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.398449732s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:16.093783686 +0000 UTC m=+3737.274926341" watchObservedRunningTime="2026-09-16 13:50:16.095678537 +0000 UTC m=+3737.276821189"
Sep 16 13:50:16.793688 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:16.793654    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:21.794087 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:21.794054    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:50:21.795907 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:21.795876    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:50:22.406044 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:22.405999    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:50:31.794597 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:31.794539    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:50:32.405598 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:32.405556    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:50:41.794239 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:41.794189    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:50:42.405921 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:42.405876    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:50:51.793739 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:51.793622    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:50:52.406265 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:50:52.406219    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:51:01.794047 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:01.793994    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:51:02.405288 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:02.405237    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:51:11.794476 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:11.794428    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:51:12.406209 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:12.406162    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:51:14.410618 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:14.410576    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:21.793854 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:21.793810    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:51:22.415392 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:22.415364    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:51:22.423186 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:22.423161    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:51:24.105419 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:24.105380    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"]
Sep 16 13:51:24.116092 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:24.116066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:26.245862 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:26.245821    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:31.794729 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:31.794687    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:51:39.131081 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:39.131023    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" containerID="cri-o://82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151" gracePeriod=45
Sep 16 13:51:39.416614 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:39.416530    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 16 13:51:40.081209 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.081179    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:51:40.172247 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172155    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-model-cache\") pod \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") "
Sep 16 13:51:40.172738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172250    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwr5g\" (UniqueName: \"kubernetes.io/projected/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kube-api-access-jwr5g\") pod \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") "
Sep 16 13:51:40.172738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172335    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-dshm\") pod \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") "
Sep 16 13:51:40.172738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172421    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-home\") pod \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") "
Sep 16 13:51:40.172738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172454    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tls-certs\") pod \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") "
Sep 16 13:51:40.172738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172461    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" (UID: "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:40.172738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172492    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-cabundle-cert\") pod \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") "
Sep 16 13:51:40.172738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172548    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tmp-dir\") pod \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") "
Sep 16 13:51:40.172738 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172574    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kserve-provision-location\") pod \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\" (UID: \"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c\") "
Sep 16 13:51:40.173350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172864    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:51:40.173350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.172912    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" (UID: "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:51:40.173350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.173202    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-home" (OuterVolumeSpecName: "home") pod "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" (UID: "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:40.175243 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.175215    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kube-api-access-jwr5g" (OuterVolumeSpecName: "kube-api-access-jwr5g") pod "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" (UID: "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c"). InnerVolumeSpecName "kube-api-access-jwr5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:51:40.175385 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.175243    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" (UID: "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:51:40.175624 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.175602    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-dshm" (OuterVolumeSpecName: "dshm") pod "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" (UID: "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:40.190865 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.190817    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" (UID: "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:40.199114 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.199088    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" (UID: "8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:40.273525 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.273495    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:51:40.273525 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.273520    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:51:40.273733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.273531    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:51:40.273733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.273544    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:51:40.273733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.273555    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:51:40.273733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.273566    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwr5g\" (UniqueName: \"kubernetes.io/projected/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-kube-api-access-jwr5g\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:51:40.273733 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.273579    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:51:40.610542 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.610513    2575 generic.go:356] "Generic (PLEG): container finished" podID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerID="82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151" exitCode=0
Sep 16 13:51:40.610741 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.610590    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"
Sep 16 13:51:40.610741 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.610608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" event={"ID":"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c","Type":"ContainerDied","Data":"82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151"}
Sep 16 13:51:40.610741 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.610707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b" event={"ID":"8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c","Type":"ContainerDied","Data":"65f32ff3a9c83a2d113d89a37c3d6ca585ecb6396b2fa7e1e9849e1e0c5450d7"}
Sep 16 13:51:40.610741 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.610738    2575 scope.go:117] "RemoveContainer" containerID="82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151"
Sep 16 13:51:40.623198 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.623179    2575 scope.go:117] "RemoveContainer" containerID="d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f"
Sep 16 13:51:40.634006 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.633986    2575 scope.go:117] "RemoveContainer" containerID="82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151"
Sep 16 13:51:40.634282 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:51:40.634257    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151\": container with ID starting with 82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151 not found: ID does not exist" containerID="82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151"
Sep 16 13:51:40.634371 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.634287    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151"} err="failed to get container status \"82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151\": rpc error: code = NotFound desc = could not find container \"82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151\": container with ID starting with 82f958409ab881b05d17a20f1a84be4ff62f2a9e6b15f46d2636c10baae36151 not found: ID does not exist"
Sep 16 13:51:40.634371 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.634304    2575 scope.go:117] "RemoveContainer" containerID="d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f"
Sep 16 13:51:40.634552 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:51:40.634537    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f\": container with ID starting with d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f not found: ID does not exist" containerID="d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f"
Sep 16 13:51:40.634595 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.634555    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f"} err="failed to get container status \"d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f\": rpc error: code = NotFound desc = could not find container \"d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f\": container with ID starting with d665ae8f4e24ef6a97b21a4ae79c0d1ab32a2b5ce2b7f92c8b4eadabbb617b7f not found: ID does not exist"
Sep 16 13:51:40.642697 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.642619    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"]
Sep 16 13:51:40.647235 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:40.647214    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-68fc8f9464-m682b"]
Sep 16 13:51:41.329340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:41.329308    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" path="/var/lib/kubelet/pods/8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c/volumes"
Sep 16 13:51:41.794781 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:41.794736    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:51:51.794273 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:51:51.794231    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:52:01.794584 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:01.794539    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:52:11.804157 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:11.804123    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:52:11.812331 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:11.812305    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:52:13.808270 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:13.808233    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:13.916996 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:13.916948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:14.021701 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:14.021668    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:20.816686 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:20.816581    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:26.819230 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:26.819192    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:26.919387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:26.919356    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:27.021897 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:27.021864    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:28.453002 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:28.452967    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:28.454745 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:28.454718    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"]
Sep 16 13:52:28.552909 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:28.552862    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:34.708111 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:34.708079    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:43.478461 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:43.478400    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" containerID="cri-o://b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19" gracePeriod=45
Sep 16 13:52:43.804966 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:43.804874    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 13:52:44.140593 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.140571    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:52:44.206982 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.206954    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7t7v\" (UniqueName: \"kubernetes.io/projected/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kube-api-access-t7t7v\") pod \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") "
Sep 16 13:52:44.207116 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207023    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tls-certs\") pod \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") "
Sep 16 13:52:44.207116 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207051    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-dshm\") pod \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") "
Sep 16 13:52:44.207116 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207079    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tmp-dir\") pod \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") "
Sep 16 13:52:44.207116 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207110    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-model-cache\") pod \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") "
Sep 16 13:52:44.207343 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207134    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-home\") pod \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") "
Sep 16 13:52:44.207343 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207168    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-cabundle-cert\") pod \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") "
Sep 16 13:52:44.207343 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207196    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kserve-provision-location\") pod \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\" (UID: \"6c2c6dcf-88cc-4991-8683-63f590a7d3e9\") "
Sep 16 13:52:44.207501 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207363    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c2c6dcf-88cc-4991-8683-63f590a7d3e9" (UID: "6c2c6dcf-88cc-4991-8683-63f590a7d3e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:52:44.207817 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207734    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c2c6dcf-88cc-4991-8683-63f590a7d3e9" (UID: "6c2c6dcf-88cc-4991-8683-63f590a7d3e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:52:44.207926 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.207874    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-home" (OuterVolumeSpecName: "home") pod "6c2c6dcf-88cc-4991-8683-63f590a7d3e9" (UID: "6c2c6dcf-88cc-4991-8683-63f590a7d3e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:52:44.209434 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.209412    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kube-api-access-t7t7v" (OuterVolumeSpecName: "kube-api-access-t7t7v") pod "6c2c6dcf-88cc-4991-8683-63f590a7d3e9" (UID: "6c2c6dcf-88cc-4991-8683-63f590a7d3e9"). InnerVolumeSpecName "kube-api-access-t7t7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:52:44.209975 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.209955    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c2c6dcf-88cc-4991-8683-63f590a7d3e9" (UID: "6c2c6dcf-88cc-4991-8683-63f590a7d3e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:52:44.209975 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.209970    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-dshm" (OuterVolumeSpecName: "dshm") pod "6c2c6dcf-88cc-4991-8683-63f590a7d3e9" (UID: "6c2c6dcf-88cc-4991-8683-63f590a7d3e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:52:44.222941 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.222916    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c2c6dcf-88cc-4991-8683-63f590a7d3e9" (UID: "6c2c6dcf-88cc-4991-8683-63f590a7d3e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:52:44.235285 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.235263    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c2c6dcf-88cc-4991-8683-63f590a7d3e9" (UID: "6c2c6dcf-88cc-4991-8683-63f590a7d3e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:52:44.307843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.307812    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:52:44.307843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.307838    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:52:44.307843 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.307850    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:52:44.308061 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.307860    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:52:44.308061 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.307871    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:52:44.308061 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.307881    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:52:44.308061 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.307893    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:52:44.308061 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.307905    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7t7v\" (UniqueName: \"kubernetes.io/projected/6c2c6dcf-88cc-4991-8683-63f590a7d3e9-kube-api-access-t7t7v\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:52:44.970747 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.970711    2575 generic.go:356] "Generic (PLEG): container finished" podID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerID="b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19" exitCode=0
Sep 16 13:52:44.971184 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.970763    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"
Sep 16 13:52:44.971184 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.970803    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" event={"ID":"6c2c6dcf-88cc-4991-8683-63f590a7d3e9","Type":"ContainerDied","Data":"b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19"}
Sep 16 13:52:44.971184 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.970859    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz" event={"ID":"6c2c6dcf-88cc-4991-8683-63f590a7d3e9","Type":"ContainerDied","Data":"9996a1bfe5241cc8aa343fee1f62e9bf71ccc550e55aae38a909cf69d5322486"}
Sep 16 13:52:44.971184 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.970885    2575 scope.go:117] "RemoveContainer" containerID="b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19"
Sep 16 13:52:44.984906 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.984890    2575 scope.go:117] "RemoveContainer" containerID="3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c"
Sep 16 13:52:44.993100 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.993082    2575 scope.go:117] "RemoveContainer" containerID="b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19"
Sep 16 13:52:44.993374 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:52:44.993355    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19\": container with ID starting with b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19 not found: ID does not exist" containerID="b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19"
Sep 16 13:52:44.993439 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.993382    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19"} err="failed to get container status \"b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19\": rpc error: code = NotFound desc = could not find container \"b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19\": container with ID starting with b183095fd8ce4353936e0f6ac0516979ea52356d531aa0dc9b531a73f8a0ae19 not found: ID does not exist"
Sep 16 13:52:44.993439 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.993399    2575 scope.go:117] "RemoveContainer" containerID="3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c"
Sep 16 13:52:44.993633 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:52:44.993615    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c\": container with ID starting with 3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c not found: ID does not exist" containerID="3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c"
Sep 16 13:52:44.993702 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.993639    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c"} err="failed to get container status \"3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c\": rpc error: code = NotFound desc = could not find container \"3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c\": container with ID starting with 3b7ac1ee44782faf95b0f0953845cc87e36a2895207a0c5b80155c2b1418341c not found: ID does not exist"
Sep 16 13:52:44.999919 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:44.999898    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"]
Sep 16 13:52:45.003483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:45.003461    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-77f8478c8f-bptwz"]
Sep 16 13:52:45.328375 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:52:45.328297    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" path="/var/lib/kubelet/pods/6c2c6dcf-88cc-4991-8683-63f590a7d3e9/volumes"
Sep 16 13:53:17.059805 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:17.059767    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:17.144577 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:17.144541    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:21.980237 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:21.980201    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"]
Sep 16 13:53:21.980672 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:21.980407    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy" containerID="cri-o://012de13200806f8809b4bb70caed65aafa2b6249adf8a860980a0e91572856a5" gracePeriod=30
Sep 16 13:53:22.027276 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:22.027245    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:25.702634 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.702596    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"]
Sep 16 13:53:25.703176 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703156    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main"
Sep 16 13:53:25.703222 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703182    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main"
Sep 16 13:53:25.703222 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703201    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="storage-initializer"
Sep 16 13:53:25.703222 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703210    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="storage-initializer"
Sep 16 13:53:25.703316 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703231    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="storage-initializer"
Sep 16 13:53:25.703316 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703240    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="storage-initializer"
Sep 16 13:53:25.703316 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703260    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main"
Sep 16 13:53:25.703316 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703269    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main"
Sep 16 13:53:25.703442 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703369    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad0e1c8-c7bf-4abf-a5a1-4e1d4376a70c" containerName="main"
Sep 16 13:53:25.703442 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.703382    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c2c6dcf-88cc-4991-8683-63f590a7d3e9" containerName="main"
Sep 16 13:53:25.707548 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.707521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.715126 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.715102    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:53:25.715325 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.715300    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-5pv5w\""
Sep 16 13:53:25.715853 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.715428    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 13:53:25.716001 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.715981    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 13:53:25.716198 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.716164    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 13:53:25.716675 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.716628    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 13:53:25.717240 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.717223    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-c6vr2\""
Sep 16 13:53:25.725312 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.725292    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"]
Sep 16 13:53:25.740150 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.740116    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"]
Sep 16 13:53:25.745740 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.745716    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:25.763885 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.763859    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"]
Sep 16 13:53:25.779670 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.779627    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:25.833899 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.833876    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.834042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.833916    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a1572af-a629-4c7c-b5ca-1bf029c9388f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.834042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.833985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.834042 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.834037    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.834224 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.834086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.834224 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.834145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.834224 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.834170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.834224 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.834213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjxsb\" (UniqueName: \"kubernetes.io/projected/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kube-api-access-cjxsb\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.935194 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935167    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:25.935340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.935340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935235    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:25.935340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935261    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.935340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.935340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjxsb\" (UniqueName: \"kubernetes.io/projected/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kube-api-access-cjxsb\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.935340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:25.935340 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f32ae73-34e2-43e8-90b9-70ae0356583a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:25.935788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4h9\" (UniqueName: \"kubernetes.io/projected/1f32ae73-34e2-43e8-90b9-70ae0356583a-kube-api-access-gb4h9\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:25.935788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.935788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a1572af-a629-4c7c-b5ca-1bf029c9388f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.935788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935459    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:25.935788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935482    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:25.935788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.935788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935557    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f32ae73-34e2-43e8-90b9-70ae0356583a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:25.935788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935618    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.935788 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.936199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935813    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.936199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935857    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.936199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.935996    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.936199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.936164    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a1572af-a629-4c7c-b5ca-1bf029c9388f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.937771 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.937748    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.938264 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.938244    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:25.947259 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:25.947235    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjxsb\" (UniqueName: \"kubernetes.io/projected/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kube-api-access-cjxsb\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:26.018914 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.018836    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:26.036904 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.036864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f32ae73-34e2-43e8-90b9-70ae0356583a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.036910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.036952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.036990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.037010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f32ae73-34e2-43e8-90b9-70ae0356583a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037025 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.037027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4h9\" (UniqueName: \"kubernetes.io/projected/1f32ae73-34e2-43e8-90b9-70ae0356583a-kube-api-access-gb4h9\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037311 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.037065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037311 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.037094    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.037344    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037464 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.037393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.037699 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.037630    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.038052 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.037959    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f32ae73-34e2-43e8-90b9-70ae0356583a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.038404 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.038376    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.039755 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.039736    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.040206 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.040191    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f32ae73-34e2-43e8-90b9-70ae0356583a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.059529 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.059497    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4h9\" (UniqueName: \"kubernetes.io/projected/1f32ae73-34e2-43e8-90b9-70ae0356583a-kube-api-access-gb4h9\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.061517 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.061485    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:26.306011 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.305871    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"]
Sep 16 13:53:26.308586 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:53:26.308556    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1572af_a629_4c7c_b5ca_1bf029c9388f.slice/crio-d45be130c0fe9ebe124b831d0d52d95e212b3d5586d168fd8ca0a2d2afc9c78e WatchSource:0}: Error finding container d45be130c0fe9ebe124b831d0d52d95e212b3d5586d168fd8ca0a2d2afc9c78e: Status 404 returned error can't find the container with id d45be130c0fe9ebe124b831d0d52d95e212b3d5586d168fd8ca0a2d2afc9c78e
Sep 16 13:53:26.355024 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.354999    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:26.647586 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:26.647558    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"]
Sep 16 13:53:26.653137 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:53:26.653107    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f32ae73_34e2_43e8_90b9_70ae0356583a.slice/crio-86f886def3887604b5202651c870482b6843f260ffd53ce4de13841f7a7649f2 WatchSource:0}: Error finding container 86f886def3887604b5202651c870482b6843f260ffd53ce4de13841f7a7649f2: Status 404 returned error can't find the container with id 86f886def3887604b5202651c870482b6843f260ffd53ce4de13841f7a7649f2
Sep 16 13:53:27.197421 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.197381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" event={"ID":"1f32ae73-34e2-43e8-90b9-70ae0356583a","Type":"ContainerStarted","Data":"e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354"}
Sep 16 13:53:27.197905 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.197451    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" event={"ID":"1f32ae73-34e2-43e8-90b9-70ae0356583a","Type":"ContainerStarted","Data":"86f886def3887604b5202651c870482b6843f260ffd53ce4de13841f7a7649f2"}
Sep 16 13:53:27.199178 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.199154    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" event={"ID":"7a1572af-a629-4c7c-b5ca-1bf029c9388f","Type":"ContainerStarted","Data":"4a90c8aae3ee97f45e74ded08e94a763e76462967e85e585a936fc2bc0ee6159"}
Sep 16 13:53:27.199321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.199202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:27.199321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.199228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" event={"ID":"7a1572af-a629-4c7c-b5ca-1bf029c9388f","Type":"ContainerStarted","Data":"d45be130c0fe9ebe124b831d0d52d95e212b3d5586d168fd8ca0a2d2afc9c78e"}
Sep 16 13:53:27.201487 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.201464    2575 generic.go:356] "Generic (PLEG): container finished" podID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerID="012de13200806f8809b4bb70caed65aafa2b6249adf8a860980a0e91572856a5" exitCode=0
Sep 16 13:53:27.201611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.201547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" event={"ID":"1aa5c41c-6f0a-40e3-8838-f060df5ddc16","Type":"ContainerDied","Data":"012de13200806f8809b4bb70caed65aafa2b6249adf8a860980a0e91572856a5"}
Sep 16 13:53:27.201611 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.201591    2575 scope.go:117] "RemoveContainer" containerID="3f46dcb25e51fe45a9fddb5427a8c9b6f158a027a5febb3de3ad4e9fa4f41ba6"
Sep 16 13:53:27.255936 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.255915    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:53:27.450825 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.450794    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-envoy\") pod \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") "
Sep 16 13:53:27.451007 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.450846    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-data\") pod \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") "
Sep 16 13:53:27.451007 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.450910    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-credential-socket\") pod \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") "
Sep 16 13:53:27.451007 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.450943    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrld2\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-kube-api-access-lrld2\") pod \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") "
Sep 16 13:53:27.451007 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.450981    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-socket\") pod \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") "
Sep 16 13:53:27.451225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451010    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-podinfo\") pod \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") "
Sep 16 13:53:27.451225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451086    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-token\") pod \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") "
Sep 16 13:53:27.451225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451117    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-certs\") pod \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") "
Sep 16 13:53:27.451225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451148    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istiod-ca-cert\") pod \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\" (UID: \"1aa5c41c-6f0a-40e3-8838-f060df5ddc16\") "
Sep 16 13:53:27.451225 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451159    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-data" (OuterVolumeSpecName: "istio-data") pod "1aa5c41c-6f0a-40e3-8838-f060df5ddc16" (UID: "1aa5c41c-6f0a-40e3-8838-f060df5ddc16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:53:27.451483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451361    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1aa5c41c-6f0a-40e3-8838-f060df5ddc16" (UID: "1aa5c41c-6f0a-40e3-8838-f060df5ddc16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:53:27.451483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451399    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1aa5c41c-6f0a-40e3-8838-f060df5ddc16" (UID: "1aa5c41c-6f0a-40e3-8838-f060df5ddc16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:53:27.451483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451418    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-data\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:53:27.451800 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451774    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1aa5c41c-6f0a-40e3-8838-f060df5ddc16" (UID: "1aa5c41c-6f0a-40e3-8838-f060df5ddc16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:53:27.451894 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.451830    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1aa5c41c-6f0a-40e3-8838-f060df5ddc16" (UID: "1aa5c41c-6f0a-40e3-8838-f060df5ddc16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:53:27.453684 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.453618    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1aa5c41c-6f0a-40e3-8838-f060df5ddc16" (UID: "1aa5c41c-6f0a-40e3-8838-f060df5ddc16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:53:27.453833 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.453813    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1aa5c41c-6f0a-40e3-8838-f060df5ddc16" (UID: "1aa5c41c-6f0a-40e3-8838-f060df5ddc16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:53:27.453903 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.453855    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-token" (OuterVolumeSpecName: "istio-token") pod "1aa5c41c-6f0a-40e3-8838-f060df5ddc16" (UID: "1aa5c41c-6f0a-40e3-8838-f060df5ddc16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:53:27.453964 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.453946    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-kube-api-access-lrld2" (OuterVolumeSpecName: "kube-api-access-lrld2") pod "1aa5c41c-6f0a-40e3-8838-f060df5ddc16" (UID: "1aa5c41c-6f0a-40e3-8838-f060df5ddc16"). InnerVolumeSpecName "kube-api-access-lrld2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:53:27.552575 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.552490    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-token\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:53:27.552575 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.552527    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:53:27.552575 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.552541    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istiod-ca-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:53:27.552575 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.552553    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-envoy\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:53:27.552575 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.552566    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-credential-socket\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:53:27.552863 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.552580    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrld2\" (UniqueName: \"kubernetes.io/projected/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-kube-api-access-lrld2\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:53:27.552863 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.552597    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-workload-socket\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:53:27.552863 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:27.552611    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa5c41c-6f0a-40e3-8838-f060df5ddc16-istio-podinfo\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:53:28.208796 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:28.208707    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"
Sep 16 13:53:28.208796 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:28.208743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn" event={"ID":"1aa5c41c-6f0a-40e3-8838-f060df5ddc16","Type":"ContainerDied","Data":"18ad0505e449f8e9d1587825d7165672b72779cb41fdccac69d41ed86a37b190"}
Sep 16 13:53:28.208796 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:28.208795    2575 scope.go:117] "RemoveContainer" containerID="012de13200806f8809b4bb70caed65aafa2b6249adf8a860980a0e91572856a5"
Sep 16 13:53:28.211262 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:28.211235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" event={"ID":"7a1572af-a629-4c7c-b5ca-1bf029c9388f","Type":"ContainerStarted","Data":"fb425b00d0e475f6a0b8a9cec89d5f94e773ae10b2d5193f3b285b6d642d0dbf"}
Sep 16 13:53:28.259304 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:28.259269    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"]
Sep 16 13:53:28.264076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:28.264051    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-zn2jn"]
Sep 16 13:53:29.331451 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:29.331414    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" path="/var/lib/kubelet/pods/1aa5c41c-6f0a-40e3-8838-f060df5ddc16/volumes"
Sep 16 13:53:30.225189 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:30.225157    2575 generic.go:356] "Generic (PLEG): container finished" podID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerID="e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354" exitCode=0
Sep 16 13:53:30.225371 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:30.225239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" event={"ID":"1f32ae73-34e2-43e8-90b9-70ae0356583a","Type":"ContainerDied","Data":"e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354"}
Sep 16 13:53:30.227521 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:30.227499    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerID="fb425b00d0e475f6a0b8a9cec89d5f94e773ae10b2d5193f3b285b6d642d0dbf" exitCode=0
Sep 16 13:53:30.227619 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:30.227532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" event={"ID":"7a1572af-a629-4c7c-b5ca-1bf029c9388f","Type":"ContainerDied","Data":"fb425b00d0e475f6a0b8a9cec89d5f94e773ae10b2d5193f3b285b6d642d0dbf"}
Sep 16 13:53:31.236546 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:31.236507    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" event={"ID":"1f32ae73-34e2-43e8-90b9-70ae0356583a","Type":"ContainerStarted","Data":"cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933"}
Sep 16 13:53:31.239158 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:31.239121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" event={"ID":"7a1572af-a629-4c7c-b5ca-1bf029c9388f","Type":"ContainerStarted","Data":"8a33511e537a6e60f81b31d19f502d705ccc9d894a74a04c4b1189d124dbb807"}
Sep 16 13:53:31.268908 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:31.268851    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podStartSLOduration=3.7227653 podStartE2EDuration="6.268831076s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.546065776s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:53:31.267146794 +0000 UTC m=+3932.448289451" watchObservedRunningTime="2026-09-16 13:53:31.268831076 +0000 UTC m=+3932.449973730"
Sep 16 13:53:31.294362 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:31.294300    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podStartSLOduration=3.7408201009999997 podStartE2EDuration="6.294278453s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.553458352s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:53:31.291707393 +0000 UTC m=+3932.472850042" watchObservedRunningTime="2026-09-16 13:53:31.294278453 +0000 UTC m=+3932.475421107"
Sep 16 13:53:36.019431 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:36.019385    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:36.019853 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:36.019469    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:36.020959 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:36.020921    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:53:36.038105 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:36.038081    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:53:36.355509 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:36.355425    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:36.355509 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:36.355485    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:53:36.357300 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:36.357265    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:53:39.472950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:39.472919    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:43.280108 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:43.280064    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:44.832168 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:44.832132    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:46.020061 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:46.020012    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:53:46.355836 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:46.355748    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:53:50.564796 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:50.564699    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:56.019833 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:56.019780    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:53:56.355786 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:56.355681    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:53:59.401016 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:53:59.400982    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:05.023361 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:05.023305    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:05.085441 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:05.085407    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:06.019359 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:06.019312    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:54:06.355807 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:06.355716    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:54:14.354834 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:14.354790    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:16.019920 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:16.019859    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:54:16.355917 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:16.355816    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:54:26.019443 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:26.019391    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:54:26.356467 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:26.356380    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:54:34.481430 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:34.481388    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:34.603986 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:34.603950    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:36.019460 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:36.019413    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:54:36.356393 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:36.356300    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:54:40.937785 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:40.937746    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:41.054480 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:41.054440    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:46.020227 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:46.020173    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:54:46.355449 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:46.355362    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:54:47.219952 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:47.219914    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:56.020091 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:56.020034    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:54:56.356205 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:54:56.356113    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:55:06.019432 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:06.019390    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:55:06.356229 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:06.356137    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:55:16.019841 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:16.019791    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:55:16.356117 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:16.356021    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:55:26.020287 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:26.020179    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:55:26.356360 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:26.356270    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:55:36.019662 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:36.019604    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:55:36.355998 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:36.355912    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:55:46.019507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:46.019460    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:55:46.356537 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:46.356443    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:55:56.020018 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:56.019966    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8001/health\": dial tcp 10.133.0.91:8001: connect: connection refused"
Sep 16 13:55:56.355812 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:56.355719    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 13:55:58.992594 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:58.992561    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:59.092414 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:55:59.092379    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:06.029580 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:06.029544    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:56:06.041038 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:06.041015    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:56:06.365258 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:06.365206    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:56:06.373274 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:06.373251    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:56:15.293215 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:15.293183    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:15.334342 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:15.334299    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"]
Sep 16 13:56:15.342133 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:15.342106    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"]
Sep 16 13:56:15.387801 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:15.387766    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:30.360437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:30.360397    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main" containerID="cri-o://cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933" gracePeriod=45
Sep 16 13:56:30.364725 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:30.364695    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main" containerID="cri-o://8a33511e537a6e60f81b31d19f502d705ccc9d894a74a04c4b1189d124dbb807" gracePeriod=45
Sep 16 13:56:31.190830 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.190775    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="llm-d-routing-sidecar" containerID="cri-o://4a90c8aae3ee97f45e74ded08e94a763e76462967e85e585a936fc2bc0ee6159" gracePeriod=45
Sep 16 13:56:31.308685 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.308635    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:56:31.325338 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.325313    2575 generic.go:356] "Generic (PLEG): container finished" podID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerID="cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933" exitCode=0
Sep 16 13:56:31.325466 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.325440    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"
Sep 16 13:56:31.328196 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.328172    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerID="8a33511e537a6e60f81b31d19f502d705ccc9d894a74a04c4b1189d124dbb807" exitCode=0
Sep 16 13:56:31.328196 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.328195    2575 generic.go:356] "Generic (PLEG): container finished" podID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerID="4a90c8aae3ee97f45e74ded08e94a763e76462967e85e585a936fc2bc0ee6159" exitCode=0
Sep 16 13:56:31.331553 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.331531    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" event={"ID":"1f32ae73-34e2-43e8-90b9-70ae0356583a","Type":"ContainerDied","Data":"cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933"}
Sep 16 13:56:31.331633 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.331588    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75" event={"ID":"1f32ae73-34e2-43e8-90b9-70ae0356583a","Type":"ContainerDied","Data":"86f886def3887604b5202651c870482b6843f260ffd53ce4de13841f7a7649f2"}
Sep 16 13:56:31.331633 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.331615    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" event={"ID":"7a1572af-a629-4c7c-b5ca-1bf029c9388f","Type":"ContainerDied","Data":"8a33511e537a6e60f81b31d19f502d705ccc9d894a74a04c4b1189d124dbb807"}
Sep 16 13:56:31.331727 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.331662    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" event={"ID":"7a1572af-a629-4c7c-b5ca-1bf029c9388f","Type":"ContainerDied","Data":"4a90c8aae3ee97f45e74ded08e94a763e76462967e85e585a936fc2bc0ee6159"}
Sep 16 13:56:31.331727 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.331682    2575 scope.go:117] "RemoveContainer" containerID="cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933"
Sep 16 13:56:31.344424 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.344406    2575 scope.go:117] "RemoveContainer" containerID="e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354"
Sep 16 13:56:31.354548 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.354528    2575 scope.go:117] "RemoveContainer" containerID="cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933"
Sep 16 13:56:31.354909 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:56:31.354883    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933\": container with ID starting with cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933 not found: ID does not exist" containerID="cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933"
Sep 16 13:56:31.354986 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.354924    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933"} err="failed to get container status \"cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933\": rpc error: code = NotFound desc = could not find container \"cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933\": container with ID starting with cd9811408e7ef0365428e33ec4144a16b7f6a527c603bf46dcc948669a556933 not found: ID does not exist"
Sep 16 13:56:31.354986 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.354949    2575 scope.go:117] "RemoveContainer" containerID="e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354"
Sep 16 13:56:31.355256 ip-10-0-135-92 kubenswrapper[2575]: E0916 13:56:31.355236    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354\": container with ID starting with e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354 not found: ID does not exist" containerID="e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354"
Sep 16 13:56:31.355326 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.355263    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354"} err="failed to get container status \"e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354\": rpc error: code = NotFound desc = could not find container \"e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354\": container with ID starting with e76487c71aacd824d9449212843a03adf9b14be436b768b79cb31c4952b2b354 not found: ID does not exist"
Sep 16 13:56:31.429922 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.429896    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:56:31.450878 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.450853    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-dshm\") pod \"1f32ae73-34e2-43e8-90b9-70ae0356583a\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") "
Sep 16 13:56:31.450878 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.450880    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-tmp-dir\") pod \"1f32ae73-34e2-43e8-90b9-70ae0356583a\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") "
Sep 16 13:56:31.451064 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.450917    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-kserve-provision-location\") pod \"1f32ae73-34e2-43e8-90b9-70ae0356583a\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") "
Sep 16 13:56:31.451064 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.450955    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f32ae73-34e2-43e8-90b9-70ae0356583a-cabundle-cert\") pod \"1f32ae73-34e2-43e8-90b9-70ae0356583a\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") "
Sep 16 13:56:31.451064 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.450991    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb4h9\" (UniqueName: \"kubernetes.io/projected/1f32ae73-34e2-43e8-90b9-70ae0356583a-kube-api-access-gb4h9\") pod \"1f32ae73-34e2-43e8-90b9-70ae0356583a\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") "
Sep 16 13:56:31.451272 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.451076    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-model-cache\") pod \"1f32ae73-34e2-43e8-90b9-70ae0356583a\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") "
Sep 16 13:56:31.451272 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.451109    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-home\") pod \"1f32ae73-34e2-43e8-90b9-70ae0356583a\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") "
Sep 16 13:56:31.451272 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.451137    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f32ae73-34e2-43e8-90b9-70ae0356583a-tls-certs\") pod \"1f32ae73-34e2-43e8-90b9-70ae0356583a\" (UID: \"1f32ae73-34e2-43e8-90b9-70ae0356583a\") "
Sep 16 13:56:31.451434 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.451306    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f32ae73-34e2-43e8-90b9-70ae0356583a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1f32ae73-34e2-43e8-90b9-70ae0356583a" (UID: "1f32ae73-34e2-43e8-90b9-70ae0356583a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:56:31.451492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.451464    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f32ae73-34e2-43e8-90b9-70ae0356583a" (UID: "1f32ae73-34e2-43e8-90b9-70ae0356583a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.452309 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.451977    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-home" (OuterVolumeSpecName: "home") pod "1f32ae73-34e2-43e8-90b9-70ae0356583a" (UID: "1f32ae73-34e2-43e8-90b9-70ae0356583a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.452309 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.452025    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f32ae73-34e2-43e8-90b9-70ae0356583a-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.454595 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.454565    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-dshm" (OuterVolumeSpecName: "dshm") pod "1f32ae73-34e2-43e8-90b9-70ae0356583a" (UID: "1f32ae73-34e2-43e8-90b9-70ae0356583a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.455207 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.455167    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f32ae73-34e2-43e8-90b9-70ae0356583a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f32ae73-34e2-43e8-90b9-70ae0356583a" (UID: "1f32ae73-34e2-43e8-90b9-70ae0356583a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:56:31.455352 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.455285    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f32ae73-34e2-43e8-90b9-70ae0356583a-kube-api-access-gb4h9" (OuterVolumeSpecName: "kube-api-access-gb4h9") pod "1f32ae73-34e2-43e8-90b9-70ae0356583a" (UID: "1f32ae73-34e2-43e8-90b9-70ae0356583a"). InnerVolumeSpecName "kube-api-access-gb4h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:56:31.466308 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.466285    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f32ae73-34e2-43e8-90b9-70ae0356583a" (UID: "1f32ae73-34e2-43e8-90b9-70ae0356583a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.475227 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.475202    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f32ae73-34e2-43e8-90b9-70ae0356583a" (UID: "1f32ae73-34e2-43e8-90b9-70ae0356583a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.552572 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.552547    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tls-certs\") pod \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") "
Sep 16 13:56:31.552717 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.552577    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tmp-dir\") pod \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") "
Sep 16 13:56:31.552717 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.552617    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a1572af-a629-4c7c-b5ca-1bf029c9388f-cabundle-cert\") pod \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") "
Sep 16 13:56:31.552717 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.552634    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kserve-provision-location\") pod \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") "
Sep 16 13:56:31.552891 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.552766    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-model-cache\") pod \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") "
Sep 16 13:56:31.552891 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.552801    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-home\") pod \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") "
Sep 16 13:56:31.552891 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.552832    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-dshm\") pod \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") "
Sep 16 13:56:31.552891 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.552855    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjxsb\" (UniqueName: \"kubernetes.io/projected/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kube-api-access-cjxsb\") pod \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\" (UID: \"7a1572af-a629-4c7c-b5ca-1bf029c9388f\") "
Sep 16 13:56:31.553110 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553007    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a1572af-a629-4c7c-b5ca-1bf029c9388f" (UID: "7a1572af-a629-4c7c-b5ca-1bf029c9388f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.553167 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553132    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.553167 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553149    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.553167 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553163    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.553492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553178    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4h9\" (UniqueName: \"kubernetes.io/projected/1f32ae73-34e2-43e8-90b9-70ae0356583a-kube-api-access-gb4h9\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.553492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553191    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.553492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553204    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.553492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553216    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f32ae73-34e2-43e8-90b9-70ae0356583a-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.553492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553227    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f32ae73-34e2-43e8-90b9-70ae0356583a-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.553492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553258    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1572af-a629-4c7c-b5ca-1bf029c9388f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a1572af-a629-4c7c-b5ca-1bf029c9388f" (UID: "7a1572af-a629-4c7c-b5ca-1bf029c9388f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:56:31.553492 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.553461    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-home" (OuterVolumeSpecName: "home") pod "7a1572af-a629-4c7c-b5ca-1bf029c9388f" (UID: "7a1572af-a629-4c7c-b5ca-1bf029c9388f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.554936 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.554913    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a1572af-a629-4c7c-b5ca-1bf029c9388f" (UID: "7a1572af-a629-4c7c-b5ca-1bf029c9388f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:56:31.555214 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.555188    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kube-api-access-cjxsb" (OuterVolumeSpecName: "kube-api-access-cjxsb") pod "7a1572af-a629-4c7c-b5ca-1bf029c9388f" (UID: "7a1572af-a629-4c7c-b5ca-1bf029c9388f"). InnerVolumeSpecName "kube-api-access-cjxsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:56:31.555431 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.555409    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-dshm" (OuterVolumeSpecName: "dshm") pod "7a1572af-a629-4c7c-b5ca-1bf029c9388f" (UID: "7a1572af-a629-4c7c-b5ca-1bf029c9388f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.565445 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.565382    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a1572af-a629-4c7c-b5ca-1bf029c9388f" (UID: "7a1572af-a629-4c7c-b5ca-1bf029c9388f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.576483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.576462    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a1572af-a629-4c7c-b5ca-1bf029c9388f" (UID: "7a1572af-a629-4c7c-b5ca-1bf029c9388f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:31.649297 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.649271    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"]
Sep 16 13:56:31.654274 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.654253    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.654274 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.654273    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.654398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.654282    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjxsb\" (UniqueName: \"kubernetes.io/projected/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kube-api-access-cjxsb\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.654398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.654291    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.654398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.654299    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.654398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.654306    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a1572af-a629-4c7c-b5ca-1bf029c9388f-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.654398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.654314    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a1572af-a629-4c7c-b5ca-1bf029c9388f-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 13:56:31.654398 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:31.654353    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58h5r75"]
Sep 16 13:56:32.337241 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:32.337153    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj" event={"ID":"7a1572af-a629-4c7c-b5ca-1bf029c9388f","Type":"ContainerDied","Data":"d45be130c0fe9ebe124b831d0d52d95e212b3d5586d168fd8ca0a2d2afc9c78e"}
Sep 16 13:56:32.337241 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:32.337190    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"
Sep 16 13:56:32.337241 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:32.337226    2575 scope.go:117] "RemoveContainer" containerID="8a33511e537a6e60f81b31d19f502d705ccc9d894a74a04c4b1189d124dbb807"
Sep 16 13:56:32.346900 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:32.346881    2575 scope.go:117] "RemoveContainer" containerID="fb425b00d0e475f6a0b8a9cec89d5f94e773ae10b2d5193f3b285b6d642d0dbf"
Sep 16 13:56:32.354753 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:32.354734    2575 scope.go:117] "RemoveContainer" containerID="4a90c8aae3ee97f45e74ded08e94a763e76462967e85e585a936fc2bc0ee6159"
Sep 16 13:56:32.363395 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:32.363325    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"]
Sep 16 13:56:32.369552 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:32.369529    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-74bd986f8-gp8jj"]
Sep 16 13:56:33.329436 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:33.329403    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" path="/var/lib/kubelet/pods/1f32ae73-34e2-43e8-90b9-70ae0356583a/volumes"
Sep 16 13:56:33.329880 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:33.329863    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" path="/var/lib/kubelet/pods/7a1572af-a629-4c7c-b5ca-1bf029c9388f/volumes"
Sep 16 13:56:47.513772 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:56:47.513734    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:05.449960 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:05.449922    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:05.591430 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:05.591381    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:08.031936 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:08.031894    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:08.126215 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:08.126176    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:12.255479 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:12.255434    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:14.459672 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:14.459619    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:14.601339 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:14.601297    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:18.950317 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:18.950283    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:19.035248 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:19.035211    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:27.837845 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.837811    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"]
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838159    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838170    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838179    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="storage-initializer"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838185    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="storage-initializer"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838194    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838199    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838205    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="llm-d-routing-sidecar"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838210    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="llm-d-routing-sidecar"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838222    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="storage-initializer"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838227    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="storage-initializer"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838232    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838238    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838247    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main"
Sep 16 13:57:27.838284 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838252    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main"
Sep 16 13:57:27.838750 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838314    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy"
Sep 16 13:57:27.838750 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838325    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f32ae73-34e2-43e8-90b9-70ae0356583a" containerName="main"
Sep 16 13:57:27.838750 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838333    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa5c41c-6f0a-40e3-8838-f060df5ddc16" containerName="istio-proxy"
Sep 16 13:57:27.838750 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838342    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="llm-d-routing-sidecar"
Sep 16 13:57:27.838750 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.838349    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a1572af-a629-4c7c-b5ca-1bf029c9388f" containerName="main"
Sep 16 13:57:27.841609 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.841589    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.845084 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.845059    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 13:57:27.845387 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.845119    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 13:57:27.847384 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.847364    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 13:57:27.847507 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.847370    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 13:57:27.847953 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.847920    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-8hhnz\""
Sep 16 13:57:27.848089 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.848020    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:57:27.856598 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.856571    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"]
Sep 16 13:57:27.883935 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.883910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.884060 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.883965    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.884060 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.884032    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.884167 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.884071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b87fed-53cc-4524-b7db-73ec52de6700-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.884167 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.884128    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.884167 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.884155    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b87fed-53cc-4524-b7db-73ec52de6700-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.884315 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.884179    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.884315 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.884226    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7c4m\" (UniqueName: \"kubernetes.io/projected/58b87fed-53cc-4524-b7db-73ec52de6700-kube-api-access-m7c4m\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.947834 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.947807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:27.984707 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.984679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b87fed-53cc-4524-b7db-73ec52de6700-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.984804 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.984726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.984804 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.984757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b87fed-53cc-4524-b7db-73ec52de6700-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.984804 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.984784    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.984925 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.984830    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7c4m\" (UniqueName: \"kubernetes.io/projected/58b87fed-53cc-4524-b7db-73ec52de6700-kube-api-access-m7c4m\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.984925 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.984859    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.984925 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.984894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.985147 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.985123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.985243 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.985222    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.985341 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.985323    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.985449 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.985430    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.985494 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.985452    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b87fed-53cc-4524-b7db-73ec52de6700-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.985541 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.985526    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.987244 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.987224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.987370 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.987354    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b87fed-53cc-4524-b7db-73ec52de6700-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:27.996897 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:27.996872    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7c4m\" (UniqueName: \"kubernetes.io/projected/58b87fed-53cc-4524-b7db-73ec52de6700-kube-api-access-m7c4m\") pod \"custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:28.153149 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:28.153109    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:28.154592 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:28.154553    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:28.432980 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:28.432949    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"]
Sep 16 13:57:28.434396 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:57:28.434361    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b87fed_53cc_4524_b7db_73ec52de6700.slice/crio-8fb566f1e24211272c73d2879954c588e65901f7123312b086a16baff1334cfe WatchSource:0}: Error finding container 8fb566f1e24211272c73d2879954c588e65901f7123312b086a16baff1334cfe: Status 404 returned error can't find the container with id 8fb566f1e24211272c73d2879954c588e65901f7123312b086a16baff1334cfe
Sep 16 13:57:28.615028 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:28.614991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" event={"ID":"58b87fed-53cc-4524-b7db-73ec52de6700","Type":"ContainerStarted","Data":"aef4108b0b2db381c6423d8b5bfa0ab3542d78d663493b85fe85de0770670144"}
Sep 16 13:57:28.615199 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:28.615050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" event={"ID":"58b87fed-53cc-4524-b7db-73ec52de6700","Type":"ContainerStarted","Data":"8fb566f1e24211272c73d2879954c588e65901f7123312b086a16baff1334cfe"}
Sep 16 13:57:31.631947 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:31.631909    2575 generic.go:356] "Generic (PLEG): container finished" podID="58b87fed-53cc-4524-b7db-73ec52de6700" containerID="aef4108b0b2db381c6423d8b5bfa0ab3542d78d663493b85fe85de0770670144" exitCode=0
Sep 16 13:57:31.632336 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:31.631984    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" event={"ID":"58b87fed-53cc-4524-b7db-73ec52de6700","Type":"ContainerDied","Data":"aef4108b0b2db381c6423d8b5bfa0ab3542d78d663493b85fe85de0770670144"}
Sep 16 13:57:32.639614 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:32.639574    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" event={"ID":"58b87fed-53cc-4524-b7db-73ec52de6700","Type":"ContainerStarted","Data":"6fe11a46f93de8211cd2b9505f638bda5453bee3110fd17b687d88039f35c849"}
Sep 16 13:57:32.665450 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:32.665398    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podStartSLOduration=3.4469827459999998 podStartE2EDuration="5.665382621s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.218399875s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:32.662553592 +0000 UTC m=+4173.843696243" watchObservedRunningTime="2026-09-16 13:57:32.665382621 +0000 UTC m=+4173.846525274"
Sep 16 13:57:38.153321 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:38.153279    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:38.153813 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:38.153399    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:57:38.154696 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:38.154666    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:57:40.577780 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.577749    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"]
Sep 16 13:57:40.580828 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.580808    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.583499 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.583477    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 13:57:40.583627 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.583559    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-554tl\""
Sep 16 13:57:40.583874 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.583855    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 13:57:40.583989 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.583882    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 16 13:57:40.583989 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.583923    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 13:57:40.584711 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.584690    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 13:57:40.594559 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.594539    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"]
Sep 16 13:57:40.612751 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.612716    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:40.692988 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.692941    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-home\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.692988 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.692987    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-dshm\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.693259 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.693008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-tmp-dir\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.693259 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.693033    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06043258-641b-4d21-beaa-6e702e196234-cabundle-cert\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.693259 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.693059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5zgp\" (UniqueName: \"kubernetes.io/projected/06043258-641b-4d21-beaa-6e702e196234-kube-api-access-g5zgp\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.693259 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.693079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-model-cache\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.693259 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.693136    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-kserve-provision-location\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.693259 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.693177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06043258-641b-4d21-beaa-6e702e196234-tls-certs\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794224 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-dshm\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794224 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-tmp-dir\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794253    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06043258-641b-4d21-beaa-6e702e196234-cabundle-cert\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5zgp\" (UniqueName: \"kubernetes.io/projected/06043258-641b-4d21-beaa-6e702e196234-kube-api-access-g5zgp\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-model-cache\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-kserve-provision-location\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794382    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06043258-641b-4d21-beaa-6e702e196234-tls-certs\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-home\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794860 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794639    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-tmp-dir\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.794860 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.794802    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-home\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.795041 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.795015    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-model-cache\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.795147 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.795125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06043258-641b-4d21-beaa-6e702e196234-cabundle-cert\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.795247 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.795227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-kserve-provision-location\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.796858 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.796832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-dshm\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.797563 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.797539    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06043258-641b-4d21-beaa-6e702e196234-tls-certs\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.803483 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.803460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5zgp\" (UniqueName: \"kubernetes.io/projected/06043258-641b-4d21-beaa-6e702e196234-kube-api-access-g5zgp\") pod \"tls-verification-test-kserve-697f76654c-lrn8k\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:40.819413 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.819388    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:40.894437 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:40.894397    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:41.178076 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:41.178046    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"]
Sep 16 13:57:41.178468 ip-10-0-135-92 kubenswrapper[2575]: W0916 13:57:41.178439    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06043258_641b_4d21_beaa_6e702e196234.slice/crio-f13e65ff1c7e233bf67d3f30191e173eccae57fafa7f620cec5397ccfcf00303 WatchSource:0}: Error finding container f13e65ff1c7e233bf67d3f30191e173eccae57fafa7f620cec5397ccfcf00303: Status 404 returned error can't find the container with id f13e65ff1c7e233bf67d3f30191e173eccae57fafa7f620cec5397ccfcf00303
Sep 16 13:57:41.690599 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:41.690554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" event={"ID":"06043258-641b-4d21-beaa-6e702e196234","Type":"ContainerStarted","Data":"e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308"}
Sep 16 13:57:41.691020 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:41.690629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" event={"ID":"06043258-641b-4d21-beaa-6e702e196234","Type":"ContainerStarted","Data":"f13e65ff1c7e233bf67d3f30191e173eccae57fafa7f620cec5397ccfcf00303"}
Sep 16 13:57:43.702827 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:43.702794    2575 generic.go:356] "Generic (PLEG): container finished" podID="06043258-641b-4d21-beaa-6e702e196234" containerID="e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308" exitCode=0
Sep 16 13:57:43.703249 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:43.702831    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" event={"ID":"06043258-641b-4d21-beaa-6e702e196234","Type":"ContainerDied","Data":"e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308"}
Sep 16 13:57:44.712535 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:44.712500    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" event={"ID":"06043258-641b-4d21-beaa-6e702e196234","Type":"ContainerStarted","Data":"16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4"}
Sep 16 13:57:44.745719 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:44.745657    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podStartSLOduration=2.357112538 podStartE2EDuration="4.745626532s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.388513994s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:44.74276554 +0000 UTC m=+4185.923908196" watchObservedRunningTime="2026-09-16 13:57:44.745626532 +0000 UTC m=+4185.926769184"
Sep 16 13:57:44.895254 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:44.895218    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:48.153803 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:48.153764    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:57:50.894810 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:50.894759    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 13:57:50.896510 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:50.896480    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:57:53.198631 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:53.198591    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:55.188141 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:55.188099    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:58.153690 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:57:58.153614    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:58:00.895538 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:00.895484    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:58:06.070343 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:06.070311    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:08.153727 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:08.153684    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:58:10.895607 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:10.895561    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:58:18.154070 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:18.154015    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:58:20.895931 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:20.895825    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:58:23.474147 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:23.474099    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:26.525633 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:26.525595    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:26.629656 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:26.629613    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:28.153880 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:28.153832    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:58:30.895424 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:30.895379    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:58:38.154590 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:38.154546    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:58:40.895122 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:40.895073    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:58:48.154519 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:48.154475    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:58:50.895133 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:50.895082    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:58:52.239484 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:52.239446    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:58.154519 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:58:58.154469    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:59:00.895191 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:00.895153    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:59:08.154399 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:08.154353    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:59:10.895441 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:10.895394    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:59:18.154350 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:18.154311    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:59:20.894936 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:20.894890    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:59:28.154012 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:28.153970    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:59:30.895666 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:30.895599    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:59:38.154309 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:38.154262    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": dial tcp 10.133.0.93:8000: connect: connection refused"
Sep 16 13:59:40.895633 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:40.895587    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:59:48.163950 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:48.163912    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:59:48.172521 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:48.172484    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 13:59:50.895304 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:50.895202    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 13:59:59.853669 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:59.853600    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:59.887619 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:59.887584    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"]
Sep 16 13:59:59.933634 ip-10-0-135-92 kubenswrapper[2575]: I0916 13:59:59.933598    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:00.895614 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:00.895576    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 14:00:10.905326 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:10.905292    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 14:00:10.913913 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:10.913886    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 14:00:14.912253 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:14.912203    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main" containerID="cri-o://6fe11a46f93de8211cd2b9505f638bda5453bee3110fd17b687d88039f35c849" gracePeriod=45
Sep 16 14:00:15.342404 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.342316    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"]
Sep 16 14:00:15.355584 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.355537    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:15.448441 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.448401    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:15.597351 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.597282    2575 generic.go:356] "Generic (PLEG): container finished" podID="58b87fed-53cc-4524-b7db-73ec52de6700" containerID="6fe11a46f93de8211cd2b9505f638bda5453bee3110fd17b687d88039f35c849" exitCode=0
Sep 16 14:00:15.597472 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.597355    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" event={"ID":"58b87fed-53cc-4524-b7db-73ec52de6700","Type":"ContainerDied","Data":"6fe11a46f93de8211cd2b9505f638bda5453bee3110fd17b687d88039f35c849"}
Sep 16 14:00:15.656150 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.656127    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 14:00:15.743120 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743089    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b87fed-53cc-4524-b7db-73ec52de6700-cabundle-cert\") pod \"58b87fed-53cc-4524-b7db-73ec52de6700\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") "
Sep 16 14:00:15.743283 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743128    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-model-cache\") pod \"58b87fed-53cc-4524-b7db-73ec52de6700\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") "
Sep 16 14:00:15.743283 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743144    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-dshm\") pod \"58b87fed-53cc-4524-b7db-73ec52de6700\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") "
Sep 16 14:00:15.743283 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743173    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b87fed-53cc-4524-b7db-73ec52de6700-tls-certs\") pod \"58b87fed-53cc-4524-b7db-73ec52de6700\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") "
Sep 16 14:00:15.743283 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743216    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-kserve-provision-location\") pod \"58b87fed-53cc-4524-b7db-73ec52de6700\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") "
Sep 16 14:00:15.743283 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743242    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7c4m\" (UniqueName: \"kubernetes.io/projected/58b87fed-53cc-4524-b7db-73ec52de6700-kube-api-access-m7c4m\") pod \"58b87fed-53cc-4524-b7db-73ec52de6700\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") "
Sep 16 14:00:15.743557 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743297    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-tmp-dir\") pod \"58b87fed-53cc-4524-b7db-73ec52de6700\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") "
Sep 16 14:00:15.743557 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743363    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-home\") pod \"58b87fed-53cc-4524-b7db-73ec52de6700\" (UID: \"58b87fed-53cc-4524-b7db-73ec52de6700\") "
Sep 16 14:00:15.743557 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743412    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-model-cache" (OuterVolumeSpecName: "model-cache") pod "58b87fed-53cc-4524-b7db-73ec52de6700" (UID: "58b87fed-53cc-4524-b7db-73ec52de6700"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.743557 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743510    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b87fed-53cc-4524-b7db-73ec52de6700-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58b87fed-53cc-4524-b7db-73ec52de6700" (UID: "58b87fed-53cc-4524-b7db-73ec52de6700"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:00:15.743809 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743790    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58b87fed-53cc-4524-b7db-73ec52de6700-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.743855 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.743816    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.744175 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.744150    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-home" (OuterVolumeSpecName: "home") pod "58b87fed-53cc-4524-b7db-73ec52de6700" (UID: "58b87fed-53cc-4524-b7db-73ec52de6700"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.745852 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.745766    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b87fed-53cc-4524-b7db-73ec52de6700-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58b87fed-53cc-4524-b7db-73ec52de6700" (UID: "58b87fed-53cc-4524-b7db-73ec52de6700"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:00:15.745852 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.745838    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-dshm" (OuterVolumeSpecName: "dshm") pod "58b87fed-53cc-4524-b7db-73ec52de6700" (UID: "58b87fed-53cc-4524-b7db-73ec52de6700"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.746187 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.746169    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b87fed-53cc-4524-b7db-73ec52de6700-kube-api-access-m7c4m" (OuterVolumeSpecName: "kube-api-access-m7c4m") pod "58b87fed-53cc-4524-b7db-73ec52de6700" (UID: "58b87fed-53cc-4524-b7db-73ec52de6700"). InnerVolumeSpecName "kube-api-access-m7c4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:00:15.756932 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.756908    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58b87fed-53cc-4524-b7db-73ec52de6700" (UID: "58b87fed-53cc-4524-b7db-73ec52de6700"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.767562 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.767536    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58b87fed-53cc-4524-b7db-73ec52de6700" (UID: "58b87fed-53cc-4524-b7db-73ec52de6700"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:15.844467 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.844437    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.844467 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.844461    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.844467 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.844472    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b87fed-53cc-4524-b7db-73ec52de6700-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.844675 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.844483    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.844675 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.844491    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7c4m\" (UniqueName: \"kubernetes.io/projected/58b87fed-53cc-4524-b7db-73ec52de6700-kube-api-access-m7c4m\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:15.844675 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:15.844500    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b87fed-53cc-4524-b7db-73ec52de6700-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:16.604451 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:16.604417    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm" event={"ID":"58b87fed-53cc-4524-b7db-73ec52de6700","Type":"ContainerDied","Data":"8fb566f1e24211272c73d2879954c588e65901f7123312b086a16baff1334cfe"}
Sep 16 14:00:16.604451 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:16.604448    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"
Sep 16 14:00:16.604928 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:16.604477    2575 scope.go:117] "RemoveContainer" containerID="6fe11a46f93de8211cd2b9505f638bda5453bee3110fd17b687d88039f35c849"
Sep 16 14:00:16.614213 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:16.614195    2575 scope.go:117] "RemoveContainer" containerID="aef4108b0b2db381c6423d8b5bfa0ab3542d78d663493b85fe85de0770670144"
Sep 16 14:00:16.628663 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:16.628626    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"]
Sep 16 14:00:16.635362 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:16.635340    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-55d88dccd7-m5mvm"]
Sep 16 14:00:17.336601 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:17.336567    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" path="/var/lib/kubelet/pods/58b87fed-53cc-4524-b7db-73ec52de6700/volumes"
Sep 16 14:00:30.366564 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:30.366515    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" containerID="cri-o://16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4" gracePeriod=45
Sep 16 14:00:30.906191 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:30.906149    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 14:00:31.113239 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.113213    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 14:00:31.160771 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.160708    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06043258-641b-4d21-beaa-6e702e196234-tls-certs\") pod \"06043258-641b-4d21-beaa-6e702e196234\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") "
Sep 16 14:00:31.160771 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.160762    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5zgp\" (UniqueName: \"kubernetes.io/projected/06043258-641b-4d21-beaa-6e702e196234-kube-api-access-g5zgp\") pod \"06043258-641b-4d21-beaa-6e702e196234\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") "
Sep 16 14:00:31.160954 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.160791    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-home\") pod \"06043258-641b-4d21-beaa-6e702e196234\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") "
Sep 16 14:00:31.160954 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.160888    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-tmp-dir\") pod \"06043258-641b-4d21-beaa-6e702e196234\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") "
Sep 16 14:00:31.160954 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.160925    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-dshm\") pod \"06043258-641b-4d21-beaa-6e702e196234\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") "
Sep 16 14:00:31.160954 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.160949    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06043258-641b-4d21-beaa-6e702e196234-cabundle-cert\") pod \"06043258-641b-4d21-beaa-6e702e196234\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") "
Sep 16 14:00:31.161172 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.160971    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-model-cache\") pod \"06043258-641b-4d21-beaa-6e702e196234\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") "
Sep 16 14:00:31.161172 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.160996    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-kserve-provision-location\") pod \"06043258-641b-4d21-beaa-6e702e196234\" (UID: \"06043258-641b-4d21-beaa-6e702e196234\") "
Sep 16 14:00:31.161299 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.161273    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06043258-641b-4d21-beaa-6e702e196234-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06043258-641b-4d21-beaa-6e702e196234" (UID: "06043258-641b-4d21-beaa-6e702e196234"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:00:31.161639 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.161615    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-model-cache" (OuterVolumeSpecName: "model-cache") pod "06043258-641b-4d21-beaa-6e702e196234" (UID: "06043258-641b-4d21-beaa-6e702e196234"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:31.161844 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.161661    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-home" (OuterVolumeSpecName: "home") pod "06043258-641b-4d21-beaa-6e702e196234" (UID: "06043258-641b-4d21-beaa-6e702e196234"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:31.163303 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.163273    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-dshm" (OuterVolumeSpecName: "dshm") pod "06043258-641b-4d21-beaa-6e702e196234" (UID: "06043258-641b-4d21-beaa-6e702e196234"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:31.163513 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.163486    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06043258-641b-4d21-beaa-6e702e196234-kube-api-access-g5zgp" (OuterVolumeSpecName: "kube-api-access-g5zgp") pod "06043258-641b-4d21-beaa-6e702e196234" (UID: "06043258-641b-4d21-beaa-6e702e196234"). InnerVolumeSpecName "kube-api-access-g5zgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:00:31.163586 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.163512    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06043258-641b-4d21-beaa-6e702e196234-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06043258-641b-4d21-beaa-6e702e196234" (UID: "06043258-641b-4d21-beaa-6e702e196234"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:00:31.176942 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.176917    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06043258-641b-4d21-beaa-6e702e196234" (UID: "06043258-641b-4d21-beaa-6e702e196234"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:31.188425 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.188400    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06043258-641b-4d21-beaa-6e702e196234" (UID: "06043258-641b-4d21-beaa-6e702e196234"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:00:31.262187 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.262160    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06043258-641b-4d21-beaa-6e702e196234-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:31.262187 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.262186    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5zgp\" (UniqueName: \"kubernetes.io/projected/06043258-641b-4d21-beaa-6e702e196234-kube-api-access-g5zgp\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:31.262373 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.262200    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:31.262373 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.262212    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:31.262373 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.262222    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:31.262373 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.262232    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06043258-641b-4d21-beaa-6e702e196234-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:31.262373 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.262243    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:31.262373 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.262255    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06043258-641b-4d21-beaa-6e702e196234-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:00:31.679777 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.679740    2575 generic.go:356] "Generic (PLEG): container finished" podID="06043258-641b-4d21-beaa-6e702e196234" containerID="16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4" exitCode=0
Sep 16 14:00:31.680176 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.679814    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"
Sep 16 14:00:31.680176 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.679824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" event={"ID":"06043258-641b-4d21-beaa-6e702e196234","Type":"ContainerDied","Data":"16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4"}
Sep 16 14:00:31.680176 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.679880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k" event={"ID":"06043258-641b-4d21-beaa-6e702e196234","Type":"ContainerDied","Data":"f13e65ff1c7e233bf67d3f30191e173eccae57fafa7f620cec5397ccfcf00303"}
Sep 16 14:00:31.680176 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.679901    2575 scope.go:117] "RemoveContainer" containerID="16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4"
Sep 16 14:00:31.689489 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.689471    2575 scope.go:117] "RemoveContainer" containerID="e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308"
Sep 16 14:00:31.697532 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.697512    2575 scope.go:117] "RemoveContainer" containerID="16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4"
Sep 16 14:00:31.697832 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:00:31.697814    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4\": container with ID starting with 16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4 not found: ID does not exist" containerID="16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4"
Sep 16 14:00:31.697896 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.697844    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4"} err="failed to get container status \"16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4\": rpc error: code = NotFound desc = could not find container \"16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4\": container with ID starting with 16b8a299760e7b4833f98ae9f01295665d692faacfd989a3dea69cd214b84ee4 not found: ID does not exist"
Sep 16 14:00:31.697896 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.697860    2575 scope.go:117] "RemoveContainer" containerID="e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308"
Sep 16 14:00:31.698131 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:00:31.698111    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308\": container with ID starting with e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308 not found: ID does not exist" containerID="e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308"
Sep 16 14:00:31.698188 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.698140    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308"} err="failed to get container status \"e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308\": rpc error: code = NotFound desc = could not find container \"e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308\": container with ID starting with e18bc2caca47705292841131d0699298d5a2566659e5a09cf0ce355b65b81308 not found: ID does not exist"
Sep 16 14:00:31.704225 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.704202    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"]
Sep 16 14:00:31.711193 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:31.711172    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-697f76654c-lrn8k"]
Sep 16 14:00:33.328165 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:33.328131    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06043258-641b-4d21-beaa-6e702e196234" path="/var/lib/kubelet/pods/06043258-641b-4d21-beaa-6e702e196234/volumes"
Sep 16 14:00:37.525084 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:37.525051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:50.016234 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.016199    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:50.109425 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.109398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:50.314038 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314008    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"]
Sep 16 14:00:50.314359 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314347    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main"
Sep 16 14:00:50.314421 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314361    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main"
Sep 16 14:00:50.314421 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314376    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="storage-initializer"
Sep 16 14:00:50.314421 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314382    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="storage-initializer"
Sep 16 14:00:50.314421 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314410    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="storage-initializer"
Sep 16 14:00:50.314421 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314416    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="storage-initializer"
Sep 16 14:00:50.314583 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314426    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main"
Sep 16 14:00:50.314583 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314432    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main"
Sep 16 14:00:50.314583 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314485    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="58b87fed-53cc-4524-b7db-73ec52de6700" containerName="main"
Sep 16 14:00:50.314583 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.314493    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="06043258-641b-4d21-beaa-6e702e196234" containerName="main"
Sep 16 14:00:50.317475 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.317456    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.326190 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.326168    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 16 14:00:50.326340 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.326321    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 14:00:50.326416 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.326396    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-x879r\""
Sep 16 14:00:50.326467 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.326428    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 14:00:50.347880 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.347853    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"]
Sep 16 14:00:50.355392 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.355366    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:50.409905 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.409881    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.410003 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.409934    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.410051 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.409998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.410051 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.410033    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd271f0-55fb-4726-92ff-5b64fd521407-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.410136 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.410058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.410136 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.410074    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.410136 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.410097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.410136 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.410114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.410315 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.410193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmlg\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-kube-api-access-tdmlg\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.442322 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.442292    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:50.511187 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511165    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511303 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511197    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511303 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511215    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511303 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511236    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmlg\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-kube-api-access-tdmlg\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511303 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511500 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511500 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511631 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd271f0-55fb-4726-92ff-5b64fd521407-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511631 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511775 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511689    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511853 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511825    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511903 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.511941 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.511877    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.512200 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.512182    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd271f0-55fb-4726-92ff-5b64fd521407-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.513804 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.513784    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.513937 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.513921    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.519686 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.519638    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmlg\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-kube-api-access-tdmlg\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.520238 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.520222    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-lwp6s\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.627977 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.627956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:50.907815 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.907792    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"]
Sep 16 14:00:50.909347 ip-10-0-135-92 kubenswrapper[2575]: W0916 14:00:50.909316    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd271f0_55fb_4726_92ff_5b64fd521407.slice/crio-6f9a468201853e7ab17f056616038ffd67fa59db30786fb3e3249472d8e42c16 WatchSource:0}: Error finding container 6f9a468201853e7ab17f056616038ffd67fa59db30786fb3e3249472d8e42c16: Status 404 returned error can't find the container with id 6f9a468201853e7ab17f056616038ffd67fa59db30786fb3e3249472d8e42c16
Sep 16 14:00:50.911312 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.911279    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:00:50.911420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.911356    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:00:50.911420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:50.911407    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:00:51.785968 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:51.785920    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" event={"ID":"6bd271f0-55fb-4726-92ff-5b64fd521407","Type":"ContainerStarted","Data":"31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec"}
Sep 16 14:00:51.786767 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:51.786741    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" event={"ID":"6bd271f0-55fb-4726-92ff-5b64fd521407","Type":"ContainerStarted","Data":"6f9a468201853e7ab17f056616038ffd67fa59db30786fb3e3249472d8e42c16"}
Sep 16 14:00:51.811077 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:51.811027    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" podStartSLOduration=1.811012227 podStartE2EDuration="1.811012227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:51.810400971 +0000 UTC m=+4372.991543625" watchObservedRunningTime="2026-09-16 14:00:51.811012227 +0000 UTC m=+4372.992154880"
Sep 16 14:00:52.628739 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:52.628702    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:52.630550 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:52.630515    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:00:53.628602 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:53.628557    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:00:53.842678 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:53.839468    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:54.629165 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:54.629110    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:00:55.628932 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:55.628874    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:00:56.629227 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:56.629182    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:00:56.887522 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:56.887420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:57.633507 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:57.633478    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:57.633980 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:57.633755    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:00:57.634450 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:00:57.634435    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:01:03.139393 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:03.139361    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:03.242375 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:03.242347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:04.289723 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.289689    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"]
Sep 16 14:01:04.294708 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.294684    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.301120 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.301093    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 14:01:04.301443 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.301241    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 14:01:04.301443 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.301105    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 14:01:04.302005 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.301566    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-rphcc\""
Sep 16 14:01:04.310537 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.310514    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"]
Sep 16 14:01:04.342354 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.342314    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:04.435866 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.435834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.436013 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.435874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.436013 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.435923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.436013 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.435970    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.436210 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.436187    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc8sp\" (UniqueName: \"kubernetes.io/projected/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kube-api-access-fc8sp\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.436280 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.436260    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.436343 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.436318    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.436408 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.436360    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-home\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.536717 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.536691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-home\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.536833 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.536731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.536833 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.536747    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.536833 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.536785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.536833 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.536807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.537045 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.536832    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8sp\" (UniqueName: \"kubernetes.io/projected/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kube-api-access-fc8sp\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.537045 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.536990    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.537045 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.537030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.537255 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.537110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-home\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.537255 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.537172    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.537372 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.537283    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.537487 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.537466    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.537665 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.537622    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.539378 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.539360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.539553 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.539537    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.545599 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.545544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc8sp\" (UniqueName: \"kubernetes.io/projected/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kube-api-access-fc8sp\") pod \"router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.605862 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.605839    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:04.782433 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.782308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:04.889895 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.889832    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"]
Sep 16 14:01:04.892763 ip-10-0-135-92 kubenswrapper[2575]: W0916 14:01:04.892734    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0624fe0b_4197_4220_bd1c_e4a5991d5e9a.slice/crio-a130bb2583452e1d2de68f7a333c2b8544ac628ae99d8a1efc51b263f4eb2a28 WatchSource:0}: Error finding container a130bb2583452e1d2de68f7a333c2b8544ac628ae99d8a1efc51b263f4eb2a28: Status 404 returned error can't find the container with id a130bb2583452e1d2de68f7a333c2b8544ac628ae99d8a1efc51b263f4eb2a28
Sep 16 14:01:04.897091 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:04.897051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:05.254354 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:05.254262    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:05.411375 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:05.411326    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:05.861723 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:05.861687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" event={"ID":"0624fe0b-4197-4220-bd1c-e4a5991d5e9a","Type":"ContainerStarted","Data":"526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d"}
Sep 16 14:01:05.861998 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:05.861976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" event={"ID":"0624fe0b-4197-4220-bd1c-e4a5991d5e9a","Type":"ContainerStarted","Data":"a130bb2583452e1d2de68f7a333c2b8544ac628ae99d8a1efc51b263f4eb2a28"}
Sep 16 14:01:07.875660 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:07.875611    2575 generic.go:356] "Generic (PLEG): container finished" podID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerID="526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d" exitCode=0
Sep 16 14:01:07.876158 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:07.875688    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" event={"ID":"0624fe0b-4197-4220-bd1c-e4a5991d5e9a","Type":"ContainerDied","Data":"526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d"}
Sep 16 14:01:08.884452 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:08.884417    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" event={"ID":"0624fe0b-4197-4220-bd1c-e4a5991d5e9a","Type":"ContainerStarted","Data":"113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379"}
Sep 16 14:01:08.914015 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:08.913964    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podStartSLOduration=2.571004296 podStartE2EDuration="4.913949068s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.342944772s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:08.910510597 +0000 UTC m=+4390.091653262" watchObservedRunningTime="2026-09-16 14:01:08.913949068 +0000 UTC m=+4390.095091722"
Sep 16 14:01:11.432896 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.432861    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"]
Sep 16 14:01:11.437777 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.437760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.444849 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.444825    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 14:01:11.445152 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.445132    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 14:01:11.445256 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.445154    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 14:01:11.445256 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.445140    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 14:01:11.445448 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.445432    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-nsrcv\""
Sep 16 14:01:11.445769 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.445755    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 14:01:11.476562 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.476538    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"]
Sep 16 14:01:11.480544 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.480522    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:11.598519 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.598484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-home\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.598519 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.598523    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tmp-dir\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.598762 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.598541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-model-cache\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.598762 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.598573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tls-certs\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.598762 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.598667    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kserve-provision-location\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.598762 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.598713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-dshm\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.598762 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.598735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p6tj\" (UniqueName: \"kubernetes.io/projected/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kube-api-access-4p6tj\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.598762 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.598759    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-cabundle-cert\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700062 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.699972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tls-certs\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700062 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700018    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kserve-provision-location\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700062 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-dshm\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700062 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p6tj\" (UniqueName: \"kubernetes.io/projected/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kube-api-access-4p6tj\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700403 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700175    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-cabundle-cert\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700403 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-home\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700403 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tmp-dir\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700403 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-model-cache\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700603 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700442    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kserve-provision-location\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700603 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700537    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-home\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700765 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700664    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tmp-dir\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.700765 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.700679    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-model-cache\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.701206 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.701132    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-cabundle-cert\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.702607 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.702577    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-dshm\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.702994 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.702974    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tls-certs\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.716852 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.716826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p6tj\" (UniqueName: \"kubernetes.io/projected/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kube-api-access-4p6tj\") pod \"prestop-hook-test-kserve-5dd855676f-ffsnp\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.748868 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.748838    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:11.780200 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:11.780167    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:12.043803 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:12.043774    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"]
Sep 16 14:01:12.045735 ip-10-0-135-92 kubenswrapper[2575]: W0916 14:01:12.045695    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30beaa9_c5ba_4f97_8f48_ca08ff8e18e3.slice/crio-f02c8e789a775882d6ab863796ba25f8bc5d11a87ffc788e3d729a1be6d0dec3 WatchSource:0}: Error finding container f02c8e789a775882d6ab863796ba25f8bc5d11a87ffc788e3d729a1be6d0dec3: Status 404 returned error can't find the container with id f02c8e789a775882d6ab863796ba25f8bc5d11a87ffc788e3d729a1be6d0dec3
Sep 16 14:01:12.906558 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:12.906517    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" event={"ID":"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3","Type":"ContainerStarted","Data":"2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586"}
Sep 16 14:01:12.907097 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:12.906583    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" event={"ID":"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3","Type":"ContainerStarted","Data":"f02c8e789a775882d6ab863796ba25f8bc5d11a87ffc788e3d729a1be6d0dec3"}
Sep 16 14:01:13.973098 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:13.973062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:14.606354 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:14.606320    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:14.606514 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:14.606390    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:01:14.607802 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:14.607768    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:01:14.918723 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:14.918686    2575 generic.go:356] "Generic (PLEG): container finished" podID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerID="2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586" exitCode=0
Sep 16 14:01:14.918875 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:14.918744    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" event={"ID":"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3","Type":"ContainerDied","Data":"2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586"}
Sep 16 14:01:15.926264 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:15.926224    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" event={"ID":"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3","Type":"ContainerStarted","Data":"0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68"}
Sep 16 14:01:15.958798 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:15.958743    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podStartSLOduration=2.548312505 podStartE2EDuration="4.95872305s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.410410545s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:15.957983923 +0000 UTC m=+4397.139126577" watchObservedRunningTime="2026-09-16 14:01:15.95872305 +0000 UTC m=+4397.139865705"
Sep 16 14:01:16.749409 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:16.749370    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:19.936370 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:19.936332    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:20.035042 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:20.035003    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:21.749205 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:21.749123    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:01:21.750592 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:21.750561    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:01:24.606736 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:24.606688    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:01:31.749426 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:31.749373    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:01:34.606632 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:34.606584    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:01:37.088596 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:37.088555    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:41.749429 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:41.749391    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:01:44.607009 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:44.606965    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:01:51.749976 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:51.749930    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:01:54.606232 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:01:54.606188    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:02:01.749874 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:01.749833    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:02:04.607080 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:04.607040    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:02:11.749964 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:11.749909    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:02:14.607270 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:14.607215    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:02:21.749624 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:21.749574    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:02:24.606616 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:24.606575    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:02:31.749821 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:31.749780    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:02:34.606461 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:34.606415    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:02:41.359868 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:41.359831    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:41.750189 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:41.750135    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:02:42.960745 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:42.960702    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:44.606582 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:44.606537    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:02:51.750058 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:51.749965    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:02:54.606445 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:02:54.606396    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:03:01.749299 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:01.749255    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:03:04.606933 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:04.606893    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:03:11.749863 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:11.749822    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:03:14.606808 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:14.606757    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:03:21.749241 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:21.749204    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:03:24.607201 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:24.607154    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/health\": dial tcp 10.133.0.96:8000: connect: connection refused"
Sep 16 14:03:31.750219 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:31.750178    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": dial tcp 10.133.0.97:8000: connect: connection refused"
Sep 16 14:03:34.618087 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:34.618051    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:03:34.626328 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:34.626293    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:03:41.759293 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:41.759259    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:03:41.766877 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:41.766854    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:03:42.250314 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.250283    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"]
Sep 16 14:03:42.277156 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.277126    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"]
Sep 16 14:03:42.283070 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.283054    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.295228 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.295196    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"]
Sep 16 14:03:42.338984 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.338950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-home\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.339105 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.338988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kserve-provision-location\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.339105 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.339009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-cabundle-cert\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.339105 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.339079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fmv\" (UniqueName: \"kubernetes.io/projected/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kube-api-access-m5fmv\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.339239 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.339134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-dshm\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.339239 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.339153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-model-cache\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.339239 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.339213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tmp-dir\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.339239 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.339232    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tls-certs\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440234 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440209    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-cabundle-cert\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440360 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fmv\" (UniqueName: \"kubernetes.io/projected/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kube-api-access-m5fmv\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440360 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-dshm\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440360 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440305    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-model-cache\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440360 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tmp-dir\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440360 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tls-certs\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440603 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-home\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440603 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440420    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kserve-provision-location\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440819 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440801    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-home\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.440869 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440848    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kserve-provision-location\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.441011 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.440994    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-cabundle-cert\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.441087 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.441013    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-model-cache\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.441087 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.441065    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tmp-dir\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.443038 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.443012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-dshm\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.443135 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.443119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tls-certs\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.448511 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.448488    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fmv\" (UniqueName: \"kubernetes.io/projected/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kube-api-access-m5fmv\") pod \"prestop-hook-test-kserve-5dd855676f-g44bw\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.594537 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.594471    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:42.877483 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:42.877458    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"]
Sep 16 14:03:42.878879 ip-10-0-135-92 kubenswrapper[2575]: W0916 14:03:42.878854    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5e6133_aaf0_4dda_837b_3aaa0dbb5323.slice/crio-1af13c6c0b551fc42b028d7c5ecd005e99fdb55e70bd60909e0a222ab70e371a WatchSource:0}: Error finding container 1af13c6c0b551fc42b028d7c5ecd005e99fdb55e70bd60909e0a222ab70e371a: Status 404 returned error can't find the container with id 1af13c6c0b551fc42b028d7c5ecd005e99fdb55e70bd60909e0a222ab70e371a
Sep 16 14:03:43.824280 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:43.824235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" event={"ID":"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323","Type":"ContainerStarted","Data":"efd53870f12f95d2b5feb64cc110af6415ccd78ccf9285bcd02499a78e6141fd"}
Sep 16 14:03:43.824502 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:43.824301    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" event={"ID":"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323","Type":"ContainerStarted","Data":"1af13c6c0b551fc42b028d7c5ecd005e99fdb55e70bd60909e0a222ab70e371a"}
Sep 16 14:03:45.837221 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:45.837189    2575 generic.go:356] "Generic (PLEG): container finished" podID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerID="efd53870f12f95d2b5feb64cc110af6415ccd78ccf9285bcd02499a78e6141fd" exitCode=0
Sep 16 14:03:45.837694 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:45.837261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" event={"ID":"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323","Type":"ContainerDied","Data":"efd53870f12f95d2b5feb64cc110af6415ccd78ccf9285bcd02499a78e6141fd"}
Sep 16 14:03:46.853253 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:46.853215    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" event={"ID":"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323","Type":"ContainerStarted","Data":"97c197cb40c4b58919d57d342a153be7f31911f67bc5ebb09573623efeaeab80"}
Sep 16 14:03:46.882461 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:46.882405    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" podStartSLOduration=2.572247801 podStartE2EDuration="4.882390223s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.310142422s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:46.879402829 +0000 UTC m=+4548.060545481" watchObservedRunningTime="2026-09-16 14:03:46.882390223 +0000 UTC m=+4548.063532920"
Sep 16 14:03:47.594751 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:47.594715    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:52.595389 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:52.595354    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:03:52.596941 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:52.596908    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" podUID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 16 14:03:57.839298 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:57.839247    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main" containerID="cri-o://0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68" gracePeriod=45
Sep 16 14:03:58.507942 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.507918    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:03:58.591942 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.591850    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-home\") pod \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") "
Sep 16 14:03:58.591942 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.591895    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-dshm\") pod \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") "
Sep 16 14:03:58.591942 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.591925    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tls-certs\") pod \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") "
Sep 16 14:03:58.592218 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.591963    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p6tj\" (UniqueName: \"kubernetes.io/projected/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kube-api-access-4p6tj\") pod \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") "
Sep 16 14:03:58.592218 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.591990    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tmp-dir\") pod \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") "
Sep 16 14:03:58.592218 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.592035    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-cabundle-cert\") pod \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") "
Sep 16 14:03:58.592218 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.592086    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-model-cache\") pod \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") "
Sep 16 14:03:58.592218 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.592137    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kserve-provision-location\") pod \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\" (UID: \"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3\") "
Sep 16 14:03:58.592488 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.592451    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-home" (OuterVolumeSpecName: "home") pod "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" (UID: "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:03:58.592553 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.592517    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" (UID: "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:03:58.592637 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.592610    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" (UID: "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:03:58.593120 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.593049    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:03:58.593275 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.593261    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:03:58.593371 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.593358    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:03:58.595151 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.594963    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" (UID: "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:03:58.595151 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.595059    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-dshm" (OuterVolumeSpecName: "dshm") pod "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" (UID: "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:03:58.595151 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.595140    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kube-api-access-4p6tj" (OuterVolumeSpecName: "kube-api-access-4p6tj") pod "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" (UID: "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3"). InnerVolumeSpecName "kube-api-access-4p6tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:03:58.611782 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.611749    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" (UID: "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:03:58.614742 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.614720    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" (UID: "f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:03:58.694339 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.694310    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:03:58.694339 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.694335    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:03:58.694339 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.694345    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:03:58.694563 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.694353    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p6tj\" (UniqueName: \"kubernetes.io/projected/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-kube-api-access-4p6tj\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:03:58.694563 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.694362    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:03:58.924139 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.924102    2575 generic.go:356] "Generic (PLEG): container finished" podID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerID="0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68" exitCode=0
Sep 16 14:03:58.924600 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.924177    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"
Sep 16 14:03:58.924600 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.924203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" event={"ID":"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3","Type":"ContainerDied","Data":"0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68"}
Sep 16 14:03:58.924600 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.924255    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp" event={"ID":"f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3","Type":"ContainerDied","Data":"f02c8e789a775882d6ab863796ba25f8bc5d11a87ffc788e3d729a1be6d0dec3"}
Sep 16 14:03:58.924600 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.924276    2575 scope.go:117] "RemoveContainer" containerID="0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68"
Sep 16 14:03:58.937382 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.937355    2575 scope.go:117] "RemoveContainer" containerID="2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586"
Sep 16 14:03:58.946333 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.946317    2575 scope.go:117] "RemoveContainer" containerID="0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68"
Sep 16 14:03:58.946576 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:03:58.946551    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68\": container with ID starting with 0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68 not found: ID does not exist" containerID="0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68"
Sep 16 14:03:58.946677 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.946589    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68"} err="failed to get container status \"0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68\": rpc error: code = NotFound desc = could not find container \"0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68\": container with ID starting with 0ee57c7614d04baa3bf1479077873a9886be3b80589e0b1ee6b6c89120827f68 not found: ID does not exist"
Sep 16 14:03:58.946677 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.946613    2575 scope.go:117] "RemoveContainer" containerID="2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586"
Sep 16 14:03:58.946928 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:03:58.946902    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586\": container with ID starting with 2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586 not found: ID does not exist" containerID="2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586"
Sep 16 14:03:58.946975 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.946940    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586"} err="failed to get container status \"2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586\": rpc error: code = NotFound desc = could not find container \"2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586\": container with ID starting with 2b4630665bcf3311ab1616365609623142283ce08d38e3ef2fd1a93fd20c5586 not found: ID does not exist"
Sep 16 14:03:58.956861 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.956839    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"]
Sep 16 14:03:58.960888 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:58.960866    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-ffsnp"]
Sep 16 14:03:59.330802 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:03:59.330729    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" path="/var/lib/kubelet/pods/f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3/volumes"
Sep 16 14:04:02.595200 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:02.595157    2575 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" podUID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Sep 16 14:04:04.517919 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:04.517869    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:04.542784 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:04.542751    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"]
Sep 16 14:04:04.586112 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:04.586072    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:05.914965 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:05.914917    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:05.939839 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:05.939803    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"]
Sep 16 14:04:05.992837 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:05.992808    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:19.566501 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:19.566456    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main" containerID="cri-o://113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379" gracePeriod=45
Sep 16 14:04:20.310195 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.310167    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:04:20.390298 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.390269    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-cabundle-cert\") pod \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") "
Sep 16 14:04:20.390476 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.390315    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-home\") pod \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") "
Sep 16 14:04:20.390476 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.390334    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-dshm\") pod \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") "
Sep 16 14:04:20.390476 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.390375    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kserve-provision-location\") pod \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") "
Sep 16 14:04:20.390476 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.390405    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tmp-dir\") pod \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") "
Sep 16 14:04:20.390476 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.390429    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-model-cache\") pod \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") "
Sep 16 14:04:20.390476 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.390455    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc8sp\" (UniqueName: \"kubernetes.io/projected/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kube-api-access-fc8sp\") pod \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") "
Sep 16 14:04:20.390476 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.390476    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tls-certs\") pod \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\" (UID: \"0624fe0b-4197-4220-bd1c-e4a5991d5e9a\") "
Sep 16 14:04:20.390906 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.390727    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0624fe0b-4197-4220-bd1c-e4a5991d5e9a" (UID: "0624fe0b-4197-4220-bd1c-e4a5991d5e9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:04:20.391055 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.391035    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0624fe0b-4197-4220-bd1c-e4a5991d5e9a" (UID: "0624fe0b-4197-4220-bd1c-e4a5991d5e9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.391149 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.391110    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-home" (OuterVolumeSpecName: "home") pod "0624fe0b-4197-4220-bd1c-e4a5991d5e9a" (UID: "0624fe0b-4197-4220-bd1c-e4a5991d5e9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.393342 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.393311    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0624fe0b-4197-4220-bd1c-e4a5991d5e9a" (UID: "0624fe0b-4197-4220-bd1c-e4a5991d5e9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:04:20.393455 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.393436    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-dshm" (OuterVolumeSpecName: "dshm") pod "0624fe0b-4197-4220-bd1c-e4a5991d5e9a" (UID: "0624fe0b-4197-4220-bd1c-e4a5991d5e9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.393455 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.393440    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kube-api-access-fc8sp" (OuterVolumeSpecName: "kube-api-access-fc8sp") pod "0624fe0b-4197-4220-bd1c-e4a5991d5e9a" (UID: "0624fe0b-4197-4220-bd1c-e4a5991d5e9a"). InnerVolumeSpecName "kube-api-access-fc8sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:04:20.401841 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.401814    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0624fe0b-4197-4220-bd1c-e4a5991d5e9a" (UID: "0624fe0b-4197-4220-bd1c-e4a5991d5e9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.425212 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.425163    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0624fe0b-4197-4220-bd1c-e4a5991d5e9a" (UID: "0624fe0b-4197-4220-bd1c-e4a5991d5e9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:20.491692 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.491612    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.491692 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.491636    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.491692 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.491659    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.491692 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.491673    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.491692 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.491686    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.491944 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.491698    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.491944 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.491711    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc8sp\" (UniqueName: \"kubernetes.io/projected/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-kube-api-access-fc8sp\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.491944 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.491718    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0624fe0b-4197-4220-bd1c-e4a5991d5e9a-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:04:20.962698 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:20.962617    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" podUID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerName="main" containerID="cri-o://97c197cb40c4b58919d57d342a153be7f31911f67bc5ebb09573623efeaeab80" gracePeriod=45
Sep 16 14:04:21.052869 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.052836    2575 generic.go:356] "Generic (PLEG): container finished" podID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerID="113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379" exitCode=0
Sep 16 14:04:21.053071 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.052892    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"
Sep 16 14:04:21.053071 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.052957    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" event={"ID":"0624fe0b-4197-4220-bd1c-e4a5991d5e9a","Type":"ContainerDied","Data":"113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379"}
Sep 16 14:04:21.053071 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.053000    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk" event={"ID":"0624fe0b-4197-4220-bd1c-e4a5991d5e9a","Type":"ContainerDied","Data":"a130bb2583452e1d2de68f7a333c2b8544ac628ae99d8a1efc51b263f4eb2a28"}
Sep 16 14:04:21.053071 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.053023    2575 scope.go:117] "RemoveContainer" containerID="113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379"
Sep 16 14:04:21.063262 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.063239    2575 scope.go:117] "RemoveContainer" containerID="526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d"
Sep 16 14:04:21.074112 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.074090    2575 scope.go:117] "RemoveContainer" containerID="113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379"
Sep 16 14:04:21.074419 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:04:21.074371    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379\": container with ID starting with 113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379 not found: ID does not exist" containerID="113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379"
Sep 16 14:04:21.074515 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.074412    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379"} err="failed to get container status \"113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379\": rpc error: code = NotFound desc = could not find container \"113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379\": container with ID starting with 113f41facde55fb450c7b503ac674e012eb0840fc3384659b51aa53ac23c5379 not found: ID does not exist"
Sep 16 14:04:21.074515 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.074436    2575 scope.go:117] "RemoveContainer" containerID="526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d"
Sep 16 14:04:21.074918 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:04:21.074889    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d\": container with ID starting with 526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d not found: ID does not exist" containerID="526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d"
Sep 16 14:04:21.075021 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.074929    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d"} err="failed to get container status \"526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d\": rpc error: code = NotFound desc = could not find container \"526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d\": container with ID starting with 526c751bda8d2eb80e9aa9c5f1ddf44e7dd981fb40986e49b5689f5ca12c652d not found: ID does not exist"
Sep 16 14:04:21.078734 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.078713    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"]
Sep 16 14:04:21.082024 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.082000    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5fb7f4f9f5-bkmnk"]
Sep 16 14:04:21.336711 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:21.336606    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" path="/var/lib/kubelet/pods/0624fe0b-4197-4220-bd1c-e4a5991d5e9a/volumes"
Sep 16 14:04:53.925453 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:53.925412    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:54.023480 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:54.023444    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:58.886577 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:58.886539    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"]
Sep 16 14:04:58.887063 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:58.886778    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerName="istio-proxy" containerID="cri-o://31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec" gracePeriod=30
Sep 16 14:04:58.981019 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:04:58.980988    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:03.891746 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:03.891707    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:04.122167 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.122132    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:04.157009 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.156944    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:05:04.278636 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278606    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-socket\") pod \"6bd271f0-55fb-4726-92ff-5b64fd521407\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") "
Sep 16 14:05:04.278860 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278689    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-data\") pod \"6bd271f0-55fb-4726-92ff-5b64fd521407\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") "
Sep 16 14:05:04.278860 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278715    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-token\") pod \"6bd271f0-55fb-4726-92ff-5b64fd521407\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") "
Sep 16 14:05:04.278860 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278736    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-podinfo\") pod \"6bd271f0-55fb-4726-92ff-5b64fd521407\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") "
Sep 16 14:05:04.278860 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278759    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd271f0-55fb-4726-92ff-5b64fd521407-istiod-ca-cert\") pod \"6bd271f0-55fb-4726-92ff-5b64fd521407\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") "
Sep 16 14:05:04.278860 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278793    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdmlg\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-kube-api-access-tdmlg\") pod \"6bd271f0-55fb-4726-92ff-5b64fd521407\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") "
Sep 16 14:05:04.278860 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278861    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-envoy\") pod \"6bd271f0-55fb-4726-92ff-5b64fd521407\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") "
Sep 16 14:05:04.279270 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278919    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-credential-socket\") pod \"6bd271f0-55fb-4726-92ff-5b64fd521407\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") "
Sep 16 14:05:04.279270 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278963    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-certs\") pod \"6bd271f0-55fb-4726-92ff-5b64fd521407\" (UID: \"6bd271f0-55fb-4726-92ff-5b64fd521407\") "
Sep 16 14:05:04.279270 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.278911    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6bd271f0-55fb-4726-92ff-5b64fd521407" (UID: "6bd271f0-55fb-4726-92ff-5b64fd521407"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:04.279270 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.279193    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6bd271f0-55fb-4726-92ff-5b64fd521407" (UID: "6bd271f0-55fb-4726-92ff-5b64fd521407"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:04.279270 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.279260    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-socket\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:04.279543 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.279022    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-data" (OuterVolumeSpecName: "istio-data") pod "6bd271f0-55fb-4726-92ff-5b64fd521407" (UID: "6bd271f0-55fb-4726-92ff-5b64fd521407"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:04.279543 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.279271    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6bd271f0-55fb-4726-92ff-5b64fd521407" (UID: "6bd271f0-55fb-4726-92ff-5b64fd521407"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:04.279865 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.279825    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd271f0-55fb-4726-92ff-5b64fd521407-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6bd271f0-55fb-4726-92ff-5b64fd521407" (UID: "6bd271f0-55fb-4726-92ff-5b64fd521407"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:05:04.281308 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.281286    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-kube-api-access-tdmlg" (OuterVolumeSpecName: "kube-api-access-tdmlg") pod "6bd271f0-55fb-4726-92ff-5b64fd521407" (UID: "6bd271f0-55fb-4726-92ff-5b64fd521407"). InnerVolumeSpecName "kube-api-access-tdmlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:04.281535 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.281499    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6bd271f0-55fb-4726-92ff-5b64fd521407" (UID: "6bd271f0-55fb-4726-92ff-5b64fd521407"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:05:04.281734 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.281665    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6bd271f0-55fb-4726-92ff-5b64fd521407" (UID: "6bd271f0-55fb-4726-92ff-5b64fd521407"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:04.282231 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.282203    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-token" (OuterVolumeSpecName: "istio-token") pod "6bd271f0-55fb-4726-92ff-5b64fd521407" (UID: "6bd271f0-55fb-4726-92ff-5b64fd521407"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:04.296630 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.296603    2575 generic.go:356] "Generic (PLEG): container finished" podID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerID="31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec" exitCode=0
Sep 16 14:05:04.296740 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.296669    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"
Sep 16 14:05:04.296740 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.296694    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" event={"ID":"6bd271f0-55fb-4726-92ff-5b64fd521407","Type":"ContainerDied","Data":"31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec"}
Sep 16 14:05:04.296816 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.296753    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s" event={"ID":"6bd271f0-55fb-4726-92ff-5b64fd521407","Type":"ContainerDied","Data":"6f9a468201853e7ab17f056616038ffd67fa59db30786fb3e3249472d8e42c16"}
Sep 16 14:05:04.296816 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.296777    2575 scope.go:117] "RemoveContainer" containerID="31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec"
Sep 16 14:05:04.308164 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.308145    2575 scope.go:117] "RemoveContainer" containerID="31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec"
Sep 16 14:05:04.308487 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:05:04.308470    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec\": container with ID starting with 31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec not found: ID does not exist" containerID="31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec"
Sep 16 14:05:04.308550 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.308496    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec"} err="failed to get container status \"31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec\": rpc error: code = NotFound desc = could not find container \"31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec\": container with ID starting with 31abe8aa096cab53c0bbe6012bdd425c5b9d31587236400327cf3aa9f286eeec not found: ID does not exist"
Sep 16 14:05:04.319524 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.319494    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"]
Sep 16 14:05:04.325003 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.324977    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-lwp6s"]
Sep 16 14:05:04.380532 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.380499    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmlg\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-kube-api-access-tdmlg\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:04.380532 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.380526    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-envoy\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:04.380532 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.380535    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-credential-socket\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:04.380767 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.380545    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-workload-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:04.380767 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.380554    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-data\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:04.380767 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.380561    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-token\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:04.380767 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.380569    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bd271f0-55fb-4726-92ff-5b64fd521407-istio-podinfo\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:04.380767 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:04.380576    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bd271f0-55fb-4726-92ff-5b64fd521407-istiod-ca-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:05.330916 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:05.330877    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" path="/var/lib/kubelet/pods/6bd271f0-55fb-4726-92ff-5b64fd521407/volumes"
Sep 16 14:05:06.310050 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.309970    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-5dd855676f-g44bw_7c5e6133-aaf0-4dda-837b-3aaa0dbb5323/main/0.log"
Sep 16 14:05:06.310399 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.310376    2575 generic.go:356] "Generic (PLEG): container finished" podID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerID="97c197cb40c4b58919d57d342a153be7f31911f67bc5ebb09573623efeaeab80" exitCode=137
Sep 16 14:05:06.310469 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.310449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" event={"ID":"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323","Type":"ContainerDied","Data":"97c197cb40c4b58919d57d342a153be7f31911f67bc5ebb09573623efeaeab80"}
Sep 16 14:05:06.615014 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.614993    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-5dd855676f-g44bw_7c5e6133-aaf0-4dda-837b-3aaa0dbb5323/main/0.log"
Sep 16 14:05:06.617545 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.617529    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:05:06.802233 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802204    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tls-certs\") pod \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") "
Sep 16 14:05:06.802233 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802239    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-model-cache\") pod \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") "
Sep 16 14:05:06.802437 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802278    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-home\") pod \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") "
Sep 16 14:05:06.802437 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802293    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kserve-provision-location\") pod \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") "
Sep 16 14:05:06.802437 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802313    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-dshm\") pod \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") "
Sep 16 14:05:06.802437 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802349    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tmp-dir\") pod \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") "
Sep 16 14:05:06.802437 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802383    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-cabundle-cert\") pod \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") "
Sep 16 14:05:06.802437 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802418    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5fmv\" (UniqueName: \"kubernetes.io/projected/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kube-api-access-m5fmv\") pod \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\" (UID: \"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323\") "
Sep 16 14:05:06.802795 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802536    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" (UID: "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:06.802795 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802623    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:06.802795 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802708    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-home" (OuterVolumeSpecName: "home") pod "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" (UID: "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:06.802947 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802931    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" (UID: "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:05:06.803006 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.802981    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" (UID: "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:06.804880 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.804851    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-dshm" (OuterVolumeSpecName: "dshm") pod "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" (UID: "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:06.805177 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.805146    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" (UID: "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:05:06.805303 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.805241    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kube-api-access-m5fmv" (OuterVolumeSpecName: "kube-api-access-m5fmv") pod "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" (UID: "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323"). InnerVolumeSpecName "kube-api-access-m5fmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:06.823130 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.823108    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" (UID: "7c5e6133-aaf0-4dda-837b-3aaa0dbb5323"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:06.903908 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.903879    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-cabundle-cert\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:06.903908 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.903904    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5fmv\" (UniqueName: \"kubernetes.io/projected/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kube-api-access-m5fmv\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:06.904043 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.903920    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:06.904043 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.903932    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:06.904043 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.903943    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-kserve-provision-location\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:06.904043 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.903954    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:06.904043 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:06.903965    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:07.318288 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:07.318210    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-5dd855676f-g44bw_7c5e6133-aaf0-4dda-837b-3aaa0dbb5323/main/0.log"
Sep 16 14:05:07.318616 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:07.318593    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"
Sep 16 14:05:07.318796 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:07.318619    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw" event={"ID":"7c5e6133-aaf0-4dda-837b-3aaa0dbb5323","Type":"ContainerDied","Data":"1af13c6c0b551fc42b028d7c5ecd005e99fdb55e70bd60909e0a222ab70e371a"}
Sep 16 14:05:07.318796 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:07.318735    2575 scope.go:117] "RemoveContainer" containerID="97c197cb40c4b58919d57d342a153be7f31911f67bc5ebb09573623efeaeab80"
Sep 16 14:05:07.329189 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:07.329160    2575 scope.go:117] "RemoveContainer" containerID="efd53870f12f95d2b5feb64cc110af6415ccd78ccf9285bcd02499a78e6141fd"
Sep 16 14:05:07.358971 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:07.358945    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"]
Sep 16 14:05:07.370800 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:07.370776    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5dd855676f-g44bw"]
Sep 16 14:05:09.328830 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:09.328785    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" path="/var/lib/kubelet/pods/7c5e6133-aaf0-4dda-837b-3aaa0dbb5323/volumes"
Sep 16 14:05:10.813489 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:10.813451    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:11.770320 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:11.770287    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:17.504288 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:17.504247    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:20.422312 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:20.422278    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:21.539240 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:21.539204    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:21.787665 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:21.787604    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:23.222136 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:23.222109    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:23.323750 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:23.323720    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:27.644876 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:27.644838    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:38.925018 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.924983    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"]
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925315    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925326    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925335    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="storage-initializer"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925341    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="storage-initializer"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925351    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="storage-initializer"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925356    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="storage-initializer"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925363    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerName="main"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925368    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerName="main"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925378    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerName="istio-proxy"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925383    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerName="istio-proxy"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925390    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925396    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925406    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerName="storage-initializer"
Sep 16 14:05:38.925420 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925411    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerName="storage-initializer"
Sep 16 14:05:38.925964 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925463    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd271f0-55fb-4726-92ff-5b64fd521407" containerName="istio-proxy"
Sep 16 14:05:38.925964 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925471    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0624fe0b-4197-4220-bd1c-e4a5991d5e9a" containerName="main"
Sep 16 14:05:38.925964 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925477    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f30beaa9-c5ba-4f97-8f48-ca08ff8e18e3" containerName="main"
Sep 16 14:05:38.925964 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.925485    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c5e6133-aaf0-4dda-837b-3aaa0dbb5323" containerName="main"
Sep 16 14:05:38.928465 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.928445    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:38.931270 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.931250    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 14:05:38.931360 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.931290    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 14:05:38.932592 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.932569    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 14:05:38.932721 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.932570    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-zsktt\""
Sep 16 14:05:38.938147 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:38.938128    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"]
Sep 16 14:05:39.045893 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.045867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.046056 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.045932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.046056 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.045979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.046056 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.046013    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.046056 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.046044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.046243 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.046060    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw55k\" (UniqueName: \"kubernetes.io/projected/b1304dae-1c8f-4117-9b83-eb8e15f219f3-kube-api-access-xw55k\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.146499 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.146475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.146618 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.146509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.146618 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.146537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.146618 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.146556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.146618 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.146590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.146618 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.146606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw55k\" (UniqueName: \"kubernetes.io/projected/b1304dae-1c8f-4117-9b83-eb8e15f219f3-kube-api-access-xw55k\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.146946 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.146921    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.147016 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.146946    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.147016 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.147002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.148908 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.148886    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.149208 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.149188    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.156476 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.156453    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw55k\" (UniqueName: \"kubernetes.io/projected/b1304dae-1c8f-4117-9b83-eb8e15f219f3-kube-api-access-xw55k\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xkz5n\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.239522 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.239453    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:39.513485 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:39.513457    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"]
Sep 16 14:05:39.515308 ip-10-0-135-92 kubenswrapper[2575]: W0916 14:05:39.515273    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1304dae_1c8f_4117_9b83_eb8e15f219f3.slice/crio-408b3f54046dd90727cf0f574cf4affa1e9f8a4343bb7c2226db19144d4e6bd0 WatchSource:0}: Error finding container 408b3f54046dd90727cf0f574cf4affa1e9f8a4343bb7c2226db19144d4e6bd0: Status 404 returned error can't find the container with id 408b3f54046dd90727cf0f574cf4affa1e9f8a4343bb7c2226db19144d4e6bd0
Sep 16 14:05:40.482956 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:40.482917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n" event={"ID":"b1304dae-1c8f-4117-9b83-eb8e15f219f3","Type":"ContainerStarted","Data":"fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69"}
Sep 16 14:05:40.483341 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:40.482978    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n" event={"ID":"b1304dae-1c8f-4117-9b83-eb8e15f219f3","Type":"ContainerStarted","Data":"408b3f54046dd90727cf0f574cf4affa1e9f8a4343bb7c2226db19144d4e6bd0"}
Sep 16 14:05:40.505841 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:40.505791    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n" podStartSLOduration=2.505775283 podStartE2EDuration="2.505775283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:05:40.503387952 +0000 UTC m=+4661.684530605" watchObservedRunningTime="2026-09-16 14:05:40.505775283 +0000 UTC m=+4661.686917936"
Sep 16 14:05:41.240247 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:41.240217    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:47.212834 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:47.212799    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:48.614041 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.614004    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:48.621194 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.621157    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"]
Sep 16 14:05:48.632117 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:05:48.632074    2575 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 14:05:48.632117 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 14:05:48.632117 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 14:05:48.632117 ip-10-0-135-92 kubenswrapper[2575]:  > containerID="fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69" cmd=["/bin/sleep","15"]
Sep 16 14:05:48.632455 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.632135    2575 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n" message="" err=<
Sep 16 14:05:48.632455 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 14:05:48.632455 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 14:05:48.632455 ip-10-0-135-92 kubenswrapper[2575]:  >
Sep 16 14:05:48.632455 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:05:48.632147    2575 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 14:05:48.632455 ip-10-0-135-92 kubenswrapper[2575]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 14:05:48.632455 ip-10-0-135-92 kubenswrapper[2575]:         , stdout: , stderr: , exit code -1
Sep 16 14:05:48.632455 ip-10-0-135-92 kubenswrapper[2575]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n" podUID="b1304dae-1c8f-4117-9b83-eb8e15f219f3" containerName="main" containerID="cri-o://fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69"
Sep 16 14:05:48.632455 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.632182    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n" podUID="b1304dae-1c8f-4117-9b83-eb8e15f219f3" containerName="main" containerID="cri-o://fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69" gracePeriod=60
Sep 16 14:05:48.679422 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.679382    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:48.878608 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.878584    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:48.924529 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.924496    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tmp-dir\") pod \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") "
Sep 16 14:05:48.924698 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.924535    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tls-certs\") pod \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") "
Sep 16 14:05:48.924698 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.924589    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-dshm\") pod \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") "
Sep 16 14:05:48.924698 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.924617    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-model-cache\") pod \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") "
Sep 16 14:05:48.924698 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.924675    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-home\") pod \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") "
Sep 16 14:05:48.924946 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.924724    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw55k\" (UniqueName: \"kubernetes.io/projected/b1304dae-1c8f-4117-9b83-eb8e15f219f3-kube-api-access-xw55k\") pod \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\" (UID: \"b1304dae-1c8f-4117-9b83-eb8e15f219f3\") "
Sep 16 14:05:48.924946 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.924794    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1304dae-1c8f-4117-9b83-eb8e15f219f3" (UID: "b1304dae-1c8f-4117-9b83-eb8e15f219f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:48.925064 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.924983    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tmp-dir\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:48.925064 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.924992    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1304dae-1c8f-4117-9b83-eb8e15f219f3" (UID: "b1304dae-1c8f-4117-9b83-eb8e15f219f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:48.925230 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.925204    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-home" (OuterVolumeSpecName: "home") pod "b1304dae-1c8f-4117-9b83-eb8e15f219f3" (UID: "b1304dae-1c8f-4117-9b83-eb8e15f219f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:48.927751 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.927720    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-dshm" (OuterVolumeSpecName: "dshm") pod "b1304dae-1c8f-4117-9b83-eb8e15f219f3" (UID: "b1304dae-1c8f-4117-9b83-eb8e15f219f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:48.927850 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.927787    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1304dae-1c8f-4117-9b83-eb8e15f219f3-kube-api-access-xw55k" (OuterVolumeSpecName: "kube-api-access-xw55k") pod "b1304dae-1c8f-4117-9b83-eb8e15f219f3" (UID: "b1304dae-1c8f-4117-9b83-eb8e15f219f3"). InnerVolumeSpecName "kube-api-access-xw55k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:48.927850 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:48.927787    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1304dae-1c8f-4117-9b83-eb8e15f219f3" (UID: "b1304dae-1c8f-4117-9b83-eb8e15f219f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:05:49.026357 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.026327    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1304dae-1c8f-4117-9b83-eb8e15f219f3-tls-certs\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.026357 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.026351    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-dshm\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.026357 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.026359    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-model-cache\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.026545 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.026367    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1304dae-1c8f-4117-9b83-eb8e15f219f3-home\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.026545 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.026375    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw55k\" (UniqueName: \"kubernetes.io/projected/b1304dae-1c8f-4117-9b83-eb8e15f219f3-kube-api-access-xw55k\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.530731 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.530640    2575 generic.go:356] "Generic (PLEG): container finished" podID="b1304dae-1c8f-4117-9b83-eb8e15f219f3" containerID="fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69" exitCode=0
Sep 16 14:05:49.530993 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.530975    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n" event={"ID":"b1304dae-1c8f-4117-9b83-eb8e15f219f3","Type":"ContainerDied","Data":"fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69"}
Sep 16 14:05:49.531164 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.531149    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n" event={"ID":"b1304dae-1c8f-4117-9b83-eb8e15f219f3","Type":"ContainerDied","Data":"408b3f54046dd90727cf0f574cf4affa1e9f8a4343bb7c2226db19144d4e6bd0"}
Sep 16 14:05:49.531295 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.531282    2575 scope.go:117] "RemoveContainer" containerID="fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69"
Sep 16 14:05:49.531614 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.531593    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"
Sep 16 14:05:49.545877 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.545858    2575 scope.go:117] "RemoveContainer" containerID="fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69"
Sep 16 14:05:49.546145 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:05:49.546124    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69\": container with ID starting with fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69 not found: ID does not exist" containerID="fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69"
Sep 16 14:05:49.546239 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.546150    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69"} err="failed to get container status \"fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69\": rpc error: code = NotFound desc = could not find container \"fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69\": container with ID starting with fe4c374dbd07d3c7906bbef1edf80515e0bdd49db5b0d2772a73a312333d5b69 not found: ID does not exist"
Sep 16 14:05:49.554780 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.554758    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"]
Sep 16 14:05:49.559606 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:49.559586    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xkz5n"]
Sep 16 14:05:51.329111 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:51.329036    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1304dae-1c8f-4117-9b83-eb8e15f219f3" path="/var/lib/kubelet/pods/b1304dae-1c8f-4117-9b83-eb8e15f219f3/volumes"
Sep 16 14:05:51.766422 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:51.766387    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:51.866925 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:05:51.866898    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:35.912001 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:35.911956    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:36.739393 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.739359    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rvlm9/must-gather-6wdx2"]
Sep 16 14:06:36.739848 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.739830    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1304dae-1c8f-4117-9b83-eb8e15f219f3" containerName="main"
Sep 16 14:06:36.739919 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.739851    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1304dae-1c8f-4117-9b83-eb8e15f219f3" containerName="main"
Sep 16 14:06:36.739919 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.739915    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1304dae-1c8f-4117-9b83-eb8e15f219f3" containerName="main"
Sep 16 14:06:36.742956 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.742936    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:06:36.746113 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.746093    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rvlm9\"/\"default-dockercfg-c2sn5\""
Sep 16 14:06:36.746412 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.746397    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvlm9\"/\"kube-root-ca.crt\""
Sep 16 14:06:36.746581 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.746564    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvlm9\"/\"openshift-service-ca.crt\""
Sep 16 14:06:36.756295 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.756271    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvlm9/must-gather-6wdx2"]
Sep 16 14:06:36.778237 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.778205    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:36.791232 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.791209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlmz\" (UniqueName: \"kubernetes.io/projected/1d74e759-7684-4bde-94bd-1739f54abd92-kube-api-access-wqlmz\") pod \"must-gather-6wdx2\" (UID: \"1d74e759-7684-4bde-94bd-1739f54abd92\") " pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:06:36.791317 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.791248    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d74e759-7684-4bde-94bd-1739f54abd92-must-gather-output\") pod \"must-gather-6wdx2\" (UID: \"1d74e759-7684-4bde-94bd-1739f54abd92\") " pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:06:36.884965 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.884939    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:36.892476 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.892454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlmz\" (UniqueName: \"kubernetes.io/projected/1d74e759-7684-4bde-94bd-1739f54abd92-kube-api-access-wqlmz\") pod \"must-gather-6wdx2\" (UID: \"1d74e759-7684-4bde-94bd-1739f54abd92\") " pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:06:36.892587 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.892485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d74e759-7684-4bde-94bd-1739f54abd92-must-gather-output\") pod \"must-gather-6wdx2\" (UID: \"1d74e759-7684-4bde-94bd-1739f54abd92\") " pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:06:36.892783 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.892769    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d74e759-7684-4bde-94bd-1739f54abd92-must-gather-output\") pod \"must-gather-6wdx2\" (UID: \"1d74e759-7684-4bde-94bd-1739f54abd92\") " pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:06:36.902326 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:36.902309    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlmz\" (UniqueName: \"kubernetes.io/projected/1d74e759-7684-4bde-94bd-1739f54abd92-kube-api-access-wqlmz\") pod \"must-gather-6wdx2\" (UID: \"1d74e759-7684-4bde-94bd-1739f54abd92\") " pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:06:37.052066 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:37.051993    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:06:37.325502 ip-10-0-135-92 kubenswrapper[2575]: W0916 14:06:37.325475    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d74e759_7684_4bde_94bd_1739f54abd92.slice/crio-ac8710a7d2298c43d3d8bcd892b1e81db214e0a17eb185ca37fc399bc492a7a9 WatchSource:0}: Error finding container ac8710a7d2298c43d3d8bcd892b1e81db214e0a17eb185ca37fc399bc492a7a9: Status 404 returned error can't find the container with id ac8710a7d2298c43d3d8bcd892b1e81db214e0a17eb185ca37fc399bc492a7a9
Sep 16 14:06:37.327740 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:37.327721    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:06:37.330961 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:37.330906    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvlm9/must-gather-6wdx2"]
Sep 16 14:06:37.776329 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:37.776297    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvlm9/must-gather-6wdx2" event={"ID":"1d74e759-7684-4bde-94bd-1739f54abd92","Type":"ContainerStarted","Data":"ac8710a7d2298c43d3d8bcd892b1e81db214e0a17eb185ca37fc399bc492a7a9"}
Sep 16 14:06:42.677187 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:42.677150    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:44.825116 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:44.825078    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvlm9/must-gather-6wdx2" event={"ID":"1d74e759-7684-4bde-94bd-1739f54abd92","Type":"ContainerStarted","Data":"556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287"}
Sep 16 14:06:44.825588 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:44.825129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvlm9/must-gather-6wdx2" event={"ID":"1d74e759-7684-4bde-94bd-1739f54abd92","Type":"ContainerStarted","Data":"39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966"}
Sep 16 14:06:44.844061 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:44.844019    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rvlm9/must-gather-6wdx2" podStartSLOduration=2.407563375 podStartE2EDuration="8.844006012s" totalImagesPullingTime="6.436442637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:06:44.841769014 +0000 UTC m=+4726.022911679" watchObservedRunningTime="2026-09-16 14:06:44.844006012 +0000 UTC m=+4726.025148664"
Sep 16 14:06:48.823505 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:48.823469    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:48.925563 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:06:48.925529    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:07:01.905382 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:01.905352    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-5swp5_0713248b-14d2-4a6b-8b5e-4dd3f5612560/manager/0.log"
Sep 16 14:07:01.932281 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:01.932253    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-vgkbm_6bd138e6-ee11-4e23-85c1-195efe4c7bbc/manager/0.log"
Sep 16 14:07:02.164328 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:02.164243    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-8stk4_8d80a30b-da72-4bcc-8997-240c5f24ac78/manager/0.log"
Sep 16 14:07:02.173880 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:02.173850    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-sd62q_1717eb9d-18fe-448a-852e-d18d95ef4b97/limitador/0.log"
Sep 16 14:07:06.952171 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:06.952139    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d74e759-7684-4bde-94bd-1739f54abd92" containerID="39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966" exitCode=0
Sep 16 14:07:06.952755 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:06.952226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvlm9/must-gather-6wdx2" event={"ID":"1d74e759-7684-4bde-94bd-1739f54abd92","Type":"ContainerDied","Data":"39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966"}
Sep 16 14:07:06.952755 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:06.952614    2575 scope.go:117] "RemoveContainer" containerID="39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966"
Sep 16 14:07:07.569034 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:07.569001    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rvlm9_must-gather-6wdx2_1d74e759-7684-4bde-94bd-1739f54abd92/gather/0.log"
Sep 16 14:07:08.163710 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.163674    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87z4h/must-gather-94896"]
Sep 16 14:07:08.166797 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.166777    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87z4h/must-gather-94896"
Sep 16 14:07:08.169452 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.169428    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87z4h\"/\"openshift-service-ca.crt\""
Sep 16 14:07:08.169568 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.169447    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87z4h\"/\"kube-root-ca.crt\""
Sep 16 14:07:08.170657 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.170628    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-87z4h\"/\"default-dockercfg-4xkqf\""
Sep 16 14:07:08.176507 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.176484    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87z4h/must-gather-94896"]
Sep 16 14:07:08.245941 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.245906    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8tx5\" (UniqueName: \"kubernetes.io/projected/08476927-e615-41ec-aa17-c98edd8f3f86-kube-api-access-z8tx5\") pod \"must-gather-94896\" (UID: \"08476927-e615-41ec-aa17-c98edd8f3f86\") " pod="openshift-must-gather-87z4h/must-gather-94896"
Sep 16 14:07:08.246051 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.245949    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08476927-e615-41ec-aa17-c98edd8f3f86-must-gather-output\") pod \"must-gather-94896\" (UID: \"08476927-e615-41ec-aa17-c98edd8f3f86\") " pod="openshift-must-gather-87z4h/must-gather-94896"
Sep 16 14:07:08.346606 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.346575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08476927-e615-41ec-aa17-c98edd8f3f86-must-gather-output\") pod \"must-gather-94896\" (UID: \"08476927-e615-41ec-aa17-c98edd8f3f86\") " pod="openshift-must-gather-87z4h/must-gather-94896"
Sep 16 14:07:08.346767 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.346709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8tx5\" (UniqueName: \"kubernetes.io/projected/08476927-e615-41ec-aa17-c98edd8f3f86-kube-api-access-z8tx5\") pod \"must-gather-94896\" (UID: \"08476927-e615-41ec-aa17-c98edd8f3f86\") " pod="openshift-must-gather-87z4h/must-gather-94896"
Sep 16 14:07:08.346908 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.346887    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08476927-e615-41ec-aa17-c98edd8f3f86-must-gather-output\") pod \"must-gather-94896\" (UID: \"08476927-e615-41ec-aa17-c98edd8f3f86\") " pod="openshift-must-gather-87z4h/must-gather-94896"
Sep 16 14:07:08.355394 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.355368    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8tx5\" (UniqueName: \"kubernetes.io/projected/08476927-e615-41ec-aa17-c98edd8f3f86-kube-api-access-z8tx5\") pod \"must-gather-94896\" (UID: \"08476927-e615-41ec-aa17-c98edd8f3f86\") " pod="openshift-must-gather-87z4h/must-gather-94896"
Sep 16 14:07:08.476543 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.476490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87z4h/must-gather-94896"
Sep 16 14:07:08.749062 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.749040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87z4h/must-gather-94896"]
Sep 16 14:07:08.751375 ip-10-0-135-92 kubenswrapper[2575]: W0916 14:07:08.751337    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08476927_e615_41ec_aa17_c98edd8f3f86.slice/crio-3e551945880e606e36bae8fbf2217c768852f4ef07dc7c7dc21b70bd404333bd WatchSource:0}: Error finding container 3e551945880e606e36bae8fbf2217c768852f4ef07dc7c7dc21b70bd404333bd: Status 404 returned error can't find the container with id 3e551945880e606e36bae8fbf2217c768852f4ef07dc7c7dc21b70bd404333bd
Sep 16 14:07:08.964873 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:08.964842    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87z4h/must-gather-94896" event={"ID":"08476927-e615-41ec-aa17-c98edd8f3f86","Type":"ContainerStarted","Data":"3e551945880e606e36bae8fbf2217c768852f4ef07dc7c7dc21b70bd404333bd"}
Sep 16 14:07:10.678701 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:10.678660    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:07:10.783701 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:10.783661    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:07:11.989383 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:11.989344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87z4h/must-gather-94896" event={"ID":"08476927-e615-41ec-aa17-c98edd8f3f86","Type":"ContainerStarted","Data":"116dadb02143be4e36a4104eee9840fe40effdc492a651d4b5f875c6983cabc7"}
Sep 16 14:07:11.989905 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:11.989676    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87z4h/must-gather-94896" event={"ID":"08476927-e615-41ec-aa17-c98edd8f3f86","Type":"ContainerStarted","Data":"97e580cee05b65285d0a93b980e5b208f84a351527f5e52a577dd43642ce7bc5"}
Sep 16 14:07:12.006764 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:12.006714    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87z4h/must-gather-94896" podStartSLOduration=1.6336849789999999 podStartE2EDuration="4.006694925s" totalImagesPullingTime="2.373009946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:12.005745875 +0000 UTC m=+4753.186888527" watchObservedRunningTime="2026-09-16 14:07:12.006694925 +0000 UTC m=+4753.187837580"
Sep 16 14:07:12.747609 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:12.747573    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ncqr5_91daa16b-8a95-4961-a53d-81125b48ceaa/global-pull-secret-syncer/0.log"
Sep 16 14:07:12.818915 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:12.818879    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log"
Sep 16 14:07:12.941444 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:12.941409    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-92.ec2.internal_377f3cc72223911dbae51e9c32e9bb13/haproxy/0.log"
Sep 16 14:07:13.028477 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.028391    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-rvlm9/must-gather-6wdx2"]
Sep 16 14:07:13.029340 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.029309    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-rvlm9/must-gather-6wdx2" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" containerName="copy" containerID="cri-o://556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287" gracePeriod=2
Sep 16 14:07:13.032148 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:07:13.032112    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6wdx2\" is forbidden: User \"system:node:ip-10-0-135-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rvlm9\": no relationship found between node 'ip-10-0-135-92.ec2.internal' and this object" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:07:13.033001 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.032978    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-rvlm9/must-gather-6wdx2"]
Sep 16 14:07:13.296715 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.296335    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rvlm9_must-gather-6wdx2_1d74e759-7684-4bde-94bd-1739f54abd92/copy/0.log"
Sep 16 14:07:13.299006 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.298984    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:07:13.302448 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:07:13.302417    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6wdx2\" is forbidden: User \"system:node:ip-10-0-135-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rvlm9\": no relationship found between node 'ip-10-0-135-92.ec2.internal' and this object" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:07:13.390272 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.390233    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d74e759-7684-4bde-94bd-1739f54abd92-must-gather-output\") pod \"1d74e759-7684-4bde-94bd-1739f54abd92\" (UID: \"1d74e759-7684-4bde-94bd-1739f54abd92\") "
Sep 16 14:07:13.390448 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.390345    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqlmz\" (UniqueName: \"kubernetes.io/projected/1d74e759-7684-4bde-94bd-1739f54abd92-kube-api-access-wqlmz\") pod \"1d74e759-7684-4bde-94bd-1739f54abd92\" (UID: \"1d74e759-7684-4bde-94bd-1739f54abd92\") "
Sep 16 14:07:13.393006 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.392977    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d74e759-7684-4bde-94bd-1739f54abd92-kube-api-access-wqlmz" (OuterVolumeSpecName: "kube-api-access-wqlmz") pod "1d74e759-7684-4bde-94bd-1739f54abd92" (UID: "1d74e759-7684-4bde-94bd-1739f54abd92"). InnerVolumeSpecName "kube-api-access-wqlmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:07:13.399470 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.399313    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d74e759-7684-4bde-94bd-1739f54abd92-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "1d74e759-7684-4bde-94bd-1739f54abd92" (UID: "1d74e759-7684-4bde-94bd-1739f54abd92"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:07:13.491095 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.491059    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqlmz\" (UniqueName: \"kubernetes.io/projected/1d74e759-7684-4bde-94bd-1739f54abd92-kube-api-access-wqlmz\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:07:13.491095 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:13.491094    2575 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d74e759-7684-4bde-94bd-1739f54abd92-must-gather-output\") on node \"ip-10-0-135-92.ec2.internal\" DevicePath \"\""
Sep 16 14:07:14.003306 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.003276    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rvlm9_must-gather-6wdx2_1d74e759-7684-4bde-94bd-1739f54abd92/copy/0.log"
Sep 16 14:07:14.003778 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.003755    2575 generic.go:356] "Generic (PLEG): container finished" podID="1d74e759-7684-4bde-94bd-1739f54abd92" containerID="556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287" exitCode=143
Sep 16 14:07:14.003894 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.003802    2575 scope.go:117] "RemoveContainer" containerID="556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287"
Sep 16 14:07:14.003894 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.003810    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvlm9/must-gather-6wdx2"
Sep 16 14:07:14.013638 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.013620    2575 scope.go:117] "RemoveContainer" containerID="39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966"
Sep 16 14:07:14.027490 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.027471    2575 scope.go:117] "RemoveContainer" containerID="556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287"
Sep 16 14:07:14.027854 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:07:14.027834    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287\": container with ID starting with 556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287 not found: ID does not exist" containerID="556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287"
Sep 16 14:07:14.027954 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.027870    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287"} err="failed to get container status \"556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287\": rpc error: code = NotFound desc = could not find container \"556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287\": container with ID starting with 556425086c39aa0f36fc68c392d34fad78d8ceca203918a8026cfbfb9cde2287 not found: ID does not exist"
Sep 16 14:07:14.027954 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.027887    2575 scope.go:117] "RemoveContainer" containerID="39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966"
Sep 16 14:07:14.028174 ip-10-0-135-92 kubenswrapper[2575]: E0916 14:07:14.028155    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966\": container with ID starting with 39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966 not found: ID does not exist" containerID="39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966"
Sep 16 14:07:14.028240 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.028179    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966"} err="failed to get container status \"39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966\": rpc error: code = NotFound desc = could not find container \"39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966\": container with ID starting with 39d1cae2d22bcbe1ace0a9f737d9bf919d7f7708300985579078c95ca4f4d966 not found: ID does not exist"
Sep 16 14:07:14.939290 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:14.939251    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:07:15.042962 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:15.042928    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:07:15.335149 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:15.335069    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" path="/var/lib/kubelet/pods/1d74e759-7684-4bde-94bd-1739f54abd92/volumes"
Sep 16 14:07:18.778305 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:18.778274    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-5swp5_0713248b-14d2-4a6b-8b5e-4dd3f5612560/manager/0.log"
Sep 16 14:07:18.810104 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:18.810037    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-vgkbm_6bd138e6-ee11-4e23-85c1-195efe4c7bbc/manager/0.log"
Sep 16 14:07:19.043721 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:19.042400    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-8stk4_8d80a30b-da72-4bcc-8997-240c5f24ac78/manager/0.log"
Sep 16 14:07:19.084241 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:19.084158    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-sd62q_1717eb9d-18fe-448a-852e-d18d95ef4b97/limitador/0.log"
Sep 16 14:07:20.927789 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:20.927690    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mh66z_937faa2a-954c-4136-8ebe-175b092bffe3/node-exporter/0.log"
Sep 16 14:07:20.949116 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:20.949080    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mh66z_937faa2a-954c-4136-8ebe-175b092bffe3/kube-rbac-proxy/0.log"
Sep 16 14:07:20.969098 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:20.969072    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mh66z_937faa2a-954c-4136-8ebe-175b092bffe3/init-textfile/0.log"
Sep 16 14:07:21.461865 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:21.461833    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf89f7648-g84th_27092ddf-a8c7-4ee2-bd2b-656cdee79acb/thanos-query/0.log"
Sep 16 14:07:21.484208 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:21.484172    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf89f7648-g84th_27092ddf-a8c7-4ee2-bd2b-656cdee79acb/kube-rbac-proxy-web/0.log"
Sep 16 14:07:21.506799 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:21.506773    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf89f7648-g84th_27092ddf-a8c7-4ee2-bd2b-656cdee79acb/kube-rbac-proxy/0.log"
Sep 16 14:07:21.535182 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:21.535148    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf89f7648-g84th_27092ddf-a8c7-4ee2-bd2b-656cdee79acb/prom-label-proxy/0.log"
Sep 16 14:07:21.559262 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:21.559232    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf89f7648-g84th_27092ddf-a8c7-4ee2-bd2b-656cdee79acb/kube-rbac-proxy-rules/0.log"
Sep 16 14:07:21.581826 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:21.581800    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cf89f7648-g84th_27092ddf-a8c7-4ee2-bd2b-656cdee79acb/kube-rbac-proxy-metrics/0.log"
Sep 16 14:07:22.430664 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.430616    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"]
Sep 16 14:07:22.431228 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.431205    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" containerName="gather"
Sep 16 14:07:22.431319 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.431230    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" containerName="gather"
Sep 16 14:07:22.431319 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.431250    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" containerName="copy"
Sep 16 14:07:22.431319 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.431259    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" containerName="copy"
Sep 16 14:07:22.431480 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.431373    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" containerName="copy"
Sep 16 14:07:22.431480 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.431389    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d74e759-7684-4bde-94bd-1739f54abd92" containerName="gather"
Sep 16 14:07:22.434269 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.434243    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.464385 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.462149    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"]
Sep 16 14:07:22.582447 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.582409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vc9\" (UniqueName: \"kubernetes.io/projected/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-kube-api-access-22vc9\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.582636 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.582454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-sys\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.582636 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.582471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-proc\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.582636 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.582564    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-podres\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.582636 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.582600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-lib-modules\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.683788 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.683695    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vc9\" (UniqueName: \"kubernetes.io/projected/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-kube-api-access-22vc9\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.684058 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.684037    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-sys\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.684172 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.684157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-proc\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.684327 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.684313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-podres\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.684423 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.684401    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-proc\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.684515 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.684501    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-lib-modules\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.684592 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.684572    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-podres\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.684678 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.684333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-sys\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.684933 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.684913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-lib-modules\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.693722 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.693681    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vc9\" (UniqueName: \"kubernetes.io/projected/445117f2-eaf8-4ef2-a364-90d9c4d4b77f-kube-api-access-22vc9\") pod \"perf-node-gather-daemonset-2g9kf\" (UID: \"445117f2-eaf8-4ef2-a364-90d9c4d4b77f\") " pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:22.756564 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:22.756537    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:23.043918 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:23.043870    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"]
Sep 16 14:07:23.078111 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:23.078077    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf" event={"ID":"445117f2-eaf8-4ef2-a364-90d9c4d4b77f","Type":"ContainerStarted","Data":"4269d60a141e61c5e850ac3a16d46bf73c581d8858cf2d3946f824d96e5de7a7"}
Sep 16 14:07:23.634703 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:23.634661    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-779f9fc776-fnwtv_597bf9e1-3371-4889-8ef0-8c7c045ee043/console/0.log"
Sep 16 14:07:24.086311 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:24.086195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf" event={"ID":"445117f2-eaf8-4ef2-a364-90d9c4d4b77f","Type":"ContainerStarted","Data":"d2ffd042fc5e761b4841c534f7fa32916712fbf16f47144678954f0c738e7712"}
Sep 16 14:07:24.087502 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:24.087475    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:24.111555 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:24.111506    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf" podStartSLOduration=2.11149195 podStartE2EDuration="2.11149195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:07:24.10784173 +0000 UTC m=+4765.288984379" watchObservedRunningTime="2026-09-16 14:07:24.11149195 +0000 UTC m=+4765.292634603"
Sep 16 14:07:24.153308 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:24.153278    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r5dpw_b8d68027-95ce-41a9-ac27-37fb01f9762c/konnectivity-agent/0.log" limit=1024
Sep 16 14:07:24.266435 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:24.266403    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-dxs4d_0ad7cdf9-0336-44ab-ac22-000333386b12/volume-data-source-validator/0.log"
Sep 16 14:07:25.369540 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:25.369505    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wfs82_497b9dd9-4306-4db6-92de-958ee5067c9a/dns/0.log"
Sep 16 14:07:25.394453 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:25.394427    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wfs82_497b9dd9-4306-4db6-92de-958ee5067c9a/kube-rbac-proxy/0.log"
Sep 16 14:07:25.418831 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:25.418802    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4p7dg_30264cd8-3720-4fbe-8ed7-cf33859e997b/dns-node-resolver/0.log"
Sep 16 14:07:26.058058 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:26.058034    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dw7rd_a61a032b-e590-47e4-b7cd-e73d1ffcf9bb/node-ca/0.log"
Sep 16 14:07:27.732551 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:27.732516    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kr5tj_0db4d710-77d0-4884-a702-739868c1c6ad/serve-healthcheck-canary/0.log"
Sep 16 14:07:28.221588 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:28.221543    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-xt4lw_6ae74286-8945-465a-9d8b-5f3ed411bade/insights-operator/1.log"
Sep 16 14:07:28.229202 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:28.229176    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-xt4lw_6ae74286-8945-465a-9d8b-5f3ed411bade/insights-operator/0.log"
Sep 16 14:07:28.253611 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:28.253583    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfjm9_63d8096b-7afb-4492-bf16-06c4c5e19de2/kube-rbac-proxy/0.log"
Sep 16 14:07:28.277713 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:28.277686    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfjm9_63d8096b-7afb-4492-bf16-06c4c5e19de2/exporter/0.log"
Sep 16 14:07:28.299198 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:28.299173    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfjm9_63d8096b-7afb-4492-bf16-06c4c5e19de2/extractor/0.log"
Sep 16 14:07:31.113671 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:31.113625    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-87z4h/perf-node-gather-daemonset-2g9kf"
Sep 16 14:07:32.052944 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:32.052913    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-867644d759-nv6h2_d0b06eed-84d3-4e40-a2cb-2748ad290ce5/manager/0.log"
Sep 16 14:07:32.601989 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:32.601904    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6c79ffff65-6qs8d_96ced9ad-3408-4138-9872-7c6d67f718b3/manager/0.log"
Sep 16 14:07:32.629559 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:32.629529    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-xh77q_8d6515d7-2d9b-423e-8ab1-2169a29017a3/server/0.log"
Sep 16 14:07:34.314674 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:34.313901    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-g4lgs_9a431196-638e-4216-ab1a-88befa2c7eb9/manager/0.log"
Sep 16 14:07:34.363055 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:34.363013    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-86zjv_96985330-40ab-4d19-b926-4638e10a21af/s3-init/0.log"
Sep 16 14:07:34.380950 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:34.380921    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-86zjv_96985330-40ab-4d19-b926-4638e10a21af/download-hf-model/0.log"
Sep 16 14:07:34.412199 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:34.412168    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-66cld_158ab7a5-b8de-46f9-8881-aabcd3cbe3e0/seaweedfs/0.log"
Sep 16 14:07:39.405139 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:39.405076    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-sfgdh_daf832f1-d96a-415c-8eb3-d6302f20b521/kube-storage-version-migrator-operator/1.log"
Sep 16 14:07:39.406726 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:39.406692    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-sfgdh_daf832f1-d96a-415c-8eb3-d6302f20b521/kube-storage-version-migrator-operator/0.log"
Sep 16 14:07:40.687753 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:40.687719    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vhhd8_8d4b5233-e0d6-49de-a7ba-d46822a74b29/kube-multus-additional-cni-plugins/0.log"
Sep 16 14:07:40.707886 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:40.707854    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vhhd8_8d4b5233-e0d6-49de-a7ba-d46822a74b29/egress-router-binary-copy/0.log"
Sep 16 14:07:40.728119 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:40.728091    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vhhd8_8d4b5233-e0d6-49de-a7ba-d46822a74b29/cni-plugins/0.log"
Sep 16 14:07:40.754095 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:40.754071    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vhhd8_8d4b5233-e0d6-49de-a7ba-d46822a74b29/bond-cni-plugin/0.log"
Sep 16 14:07:40.777777 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:40.777756    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vhhd8_8d4b5233-e0d6-49de-a7ba-d46822a74b29/routeoverride-cni/0.log"
Sep 16 14:07:40.801370 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:40.801348    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vhhd8_8d4b5233-e0d6-49de-a7ba-d46822a74b29/whereabouts-cni-bincopy/0.log"
Sep 16 14:07:40.825668 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:40.825627    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vhhd8_8d4b5233-e0d6-49de-a7ba-d46822a74b29/whereabouts-cni/0.log"
Sep 16 14:07:40.938260 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:40.938191    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pwrvh_ed8824d0-c77c-4605-8df5-8dd50e22c1fc/kube-multus/0.log"
Sep 16 14:07:41.017268 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:41.017241    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6xf6c_f60d9fd9-2504-4b8d-b9a6-257b20027e36/network-metrics-daemon/0.log"
Sep 16 14:07:41.031848 ip-10-0-135-92 kubenswrapper[2575]: I0916 14:07:41.031824    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6xf6c_f60d9fd9-2504-4b8d-b9a6-257b20027e36/kube-rbac-proxy/0.log"