Sep 16 19:37:24.588449 ip-10-0-141-26 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 19:37:24.588464 ip-10-0-141-26 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 19:37:24.588474 ip-10-0-141-26 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 19:37:24.588784 ip-10-0-141-26 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 19:37:34.685100 ip-10-0-141-26 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 19:37:34.685116 ip-10-0-141-26 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 03b49a45150e4ed1beb62f273572193b --
Sep 16 19:39:28.322715 ip-10-0-141-26 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 19:39:28.724466 ip-10-0-141-26 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:39:28.724466 ip-10-0-141-26 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 19:39:28.724466 ip-10-0-141-26 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:39:28.724466 ip-10-0-141-26 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 19:39:28.724466 ip-10-0-141-26 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 19:39:28.726294 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.726198    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 19:39:28.729155 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729137    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:28.729155 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729155    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729159    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729163    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729165    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729169    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729172    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729174    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729177    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729180    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729182    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729185    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729188    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729190    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729193    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729195    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729199    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729204    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729208    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729219    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:28.729225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729222    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729225    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729227    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729230    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729233    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729236    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729239    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729242    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729244    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729247    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729250    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729252    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729255    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729257    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729260    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729262    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729265    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729267    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729270    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:28.729695 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729272    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729275    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729277    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729280    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729282    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729285    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729287    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729290    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729294    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729296    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729299    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729301    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729304    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729306    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729308    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729311    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729314    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729318    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729320    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729323    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:28.730154 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729326    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729328    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729332    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729335    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729337    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729340    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729342    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729345    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729348    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729352    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729357    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729360    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729363    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729366    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729368    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729371    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729373    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729376    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729378    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:28.730673 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729380    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729383    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729385    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729388    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729391    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729393    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729396    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729410    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729414    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729417    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729420    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729423    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729426    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729428    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729431    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729433    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729437    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729441    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729444    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729446    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:28.731177 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.729449    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730164    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730171    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730174    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730177    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730180    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730183    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730186    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730189    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730192    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730194    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730197    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730200    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730203    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730205    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730208    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730211    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730213    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730216    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730219    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:28.731781 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730222    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730225    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730227    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730230    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730233    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730235    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730238    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730240    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730243    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730246    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730248    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730252    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730255    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730257    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730260    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730262    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730266    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730268    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730272    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:28.732284 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730275    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730277    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730280    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730282    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730285    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730288    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730291    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730293    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730296    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730298    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730300    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730303    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730305    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730308    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730311    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730314    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730316    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730321    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730324    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730327    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:28.732773 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730331    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730333    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730336    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730339    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730341    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730344    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730346    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730349    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730352    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730354    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730357    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730359    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730362    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730368    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730371    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730373    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730376    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730379    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730381    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:28.733269 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730384    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730386    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730389    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730391    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730394    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730396    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730415    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730418    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730420    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730423    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730425    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730430    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730433    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730436    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730439    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730441    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730444    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730447    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730450    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730453    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:28.733745 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730455    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730458    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730864    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730868    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730871    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730874    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730877    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730879    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730882    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730888    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730891    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730893    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730896    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730899    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730901    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730904    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730906    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730909    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730912    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730915    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:28.734234 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730918    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730920    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730923    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730926    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730928    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730931    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730933    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730936    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730939    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730943    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730945    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730948    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730950    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730953    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730955    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730958    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730960    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730963    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730965    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730969    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:28.734743 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730971    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730976    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730979    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730981    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730984    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730987    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730989    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730991    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730994    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.730998    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731000    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731002    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731005    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731007    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731010    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731012    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731015    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731017    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731022    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731025    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:28.735237 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731028    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731031    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731033    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731036    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731038    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731041    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731044    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731046    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731048    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731051    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731053    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731057    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731061    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731067    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731070    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731073    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731075    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731078    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731080    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731083    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:28.735769 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731085    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731088    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731090    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731094    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731096    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731099    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731101    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731104    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731106    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731108    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731111    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731113    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731116    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731118    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731121    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731123    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731126    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731128    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731131    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:28.736258 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731133    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731136    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731221    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731255    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731265    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731269    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731277    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731280    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731289    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731293    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731296    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731299    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731304    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731307    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731310    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731313    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731316    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731320    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731323    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731327    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731331    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731333    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731337    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731340    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 19:39:28.736750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731343    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731347    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731350    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731354    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731357    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731360    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731363    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731366    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731369    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731372    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731377    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731380    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731382    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731386    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731393    2574 flags.go:64] FLAG: --enable-server="true"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731397    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731415    2574 flags.go:64] FLAG: --event-burst="100"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731418    2574 flags.go:64] FLAG: --event-qps="50"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731421    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731425    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731427    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731432    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731434    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731437    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731441    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 16 19:39:28.737397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731444    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731447    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731450    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731453    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731456    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731459    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731462    2574 flags.go:64] FLAG: --feature-gates=""
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731466    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731469    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731472    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731475    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731478    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731481    2574 flags.go:64] FLAG: --help="false"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731484    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-141-26.ec2.internal"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731487    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731490    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731493    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731497    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731500    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731503    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731506    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731511    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731514    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 19:39:28.738023 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731518    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731521    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731524    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731527    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731530    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731533    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731536    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731539    2574 flags.go:64] FLAG: --lock-file=""
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731541    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731545    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731547    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731554    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731557    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731560    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731563    2574 flags.go:64] FLAG: --logging-format="text"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731566    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731569    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731572    2574 flags.go:64] FLAG: --manifest-url=""
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731575    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731579    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731583    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731587    2574 flags.go:64] FLAG: --max-pods="110"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731590    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731593    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731596    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 19:39:28.738670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731599    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731602    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731605    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731608    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731614    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731621    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731624    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731627    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731631    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731636    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731639    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731643    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731646    2574 flags.go:64] FLAG: --port="10250"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731649    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731652    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-011d99346fadfa119"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731655    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731658    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731661    2574 flags.go:64] FLAG: --register-node="true"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731664    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731667    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731670    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731673    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731676    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731680    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731683    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 19:39:28.739298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731686    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731690    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731692    2574 flags.go:64] FLAG: --runonce="false"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731695    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731698    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731702    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731704    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731707    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731711    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731714    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731717    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731720    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731725    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731729    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731732    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731735    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731738    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731741    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731746    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731749    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731752    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731756    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731759    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731762    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731764    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 19:39:28.739991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731767    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731771    2574 flags.go:64] FLAG: --v="2"
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731775    2574 flags.go:64] FLAG: --version="false"
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731779    2574 flags.go:64] FLAG: --vmodule=""
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731783    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.731787    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731898    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731902    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731905    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731910    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731913    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731916    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731919    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731922    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731925    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731928    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731931    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731933    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731936    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731940    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:28.740626 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731944    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731948    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731951    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731954    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731957    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731960    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731963    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731965    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731968    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731971    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731974    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731977    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731981    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731985    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731987    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731990    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731992    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731995    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.731998    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732000    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:28.741121 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732002    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732005    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732007    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732010    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732013    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732016    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732019    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732021    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732024    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732026    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732029    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732032    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732034    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732037    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732040    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732043    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732045    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732048    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732051    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732053    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:28.741621 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732056    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732058    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732061    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732063    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732066    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732078    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732081    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732084    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732087    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732089    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732092    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732095    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732098    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732100    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732103    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732106    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732109    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732112    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732114    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732117    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:28.742116 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732119    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732122    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732124    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732127    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732130    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732133    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732135    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732138    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732141    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732143    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732146    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732149    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732151    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732154    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732156    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732159    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732161    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732164    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732166    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732169    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:28.742692 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732171    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732174    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732177    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732180    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.732182    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.732188    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.740761    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.740777    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.742152    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.742165    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742227    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742232    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742236    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742239    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742242    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:28.743214 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742245    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742248    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742250    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742253    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742256    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742258    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742261    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742264    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742266    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742269    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742272    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742274    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742276    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742279    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742282    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742285    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742287    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742290    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742292    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742295    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:28.743598 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742298    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742301    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742303    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742306    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742310    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742313    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742316    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742318    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742321    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742324    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742326    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742331    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742334    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742337    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742340    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742343    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742346    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742348    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742351    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:28.744079 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742354    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742356    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742359    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742363    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742367    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742370    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742373    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742375    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742378    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742381    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742384    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742386    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742390    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742392    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742395    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742397    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742414    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742418    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742422    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742425    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:28.744559 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742427    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742430    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742432    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742435    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742437    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742440    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742443    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742445    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742448    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742450    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742453    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742457    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742461    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742463    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742466    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742469    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742472    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742475    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742478    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742480    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:28.745043 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742483    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742485    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742488    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742491    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742494    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742496    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742499    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742501    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742503    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742507    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742510    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742513    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742515    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742518    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742520    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:28.745566 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.742525    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742639    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742644    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742648    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742650    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742653    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742656    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742659    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742664    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742667    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742670    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742672    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742675    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742678    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742681    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742684    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742686    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742689    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742693    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 19:39:28.745972 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742695    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742698    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742701    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742704    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742706    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742709    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742713    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742717    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742721    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742724    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742727    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742730    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742733    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742736    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742739    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742741    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742744    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742746    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742749    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 19:39:28.746485 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742752    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742754    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742757    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742759    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742762    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742764    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742767    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742770    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742772    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742775    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742777    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742780    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742782    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742785    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742787    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742790    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742793    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742795    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742798    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742800    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 19:39:28.746948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742803    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742806    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742808    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742811    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742814    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742817    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742819    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742822    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742824    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742827    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742829    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742832    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742834    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742837    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742839    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742842    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742844    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742847    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742849    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 19:39:28.747440 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742852    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742854    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742857    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742859    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742862    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742865    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742867    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742869    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742872    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742874    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742877    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742879    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742882    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742884    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742887    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742890    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742892    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742895    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742898    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742900    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 19:39:28.747906 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742902    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 19:39:28.748394 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742905    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 19:39:28.748394 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:28.742907    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 19:39:28.748394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.742912    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 19:39:28.748394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.743023    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 19:39:28.748394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.744983    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 19:39:28.748394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.745797    2574 server.go:1019] "Starting client certificate rotation"
Sep 16 19:39:28.748394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.745900    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 19:39:28.748394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.745935    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 19:39:28.750602 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.750557    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 19:39:28.757236 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.757210    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 19:39:28.769257 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.769235    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 16 19:39:28.772130 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.772112    2574 log.go:25] "Validated CRI v1 image API"
Sep 16 19:39:28.773099 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.773081    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 19:39:28.776012 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.775976    2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ab73be23-720b-4001-af9a-845ecf12c503:/dev/nvme0n1p3 c2ad108f-b8fa-4fd2-b178-1d04728bed14:/dev/nvme0n1p4]
Sep 16 19:39:28.776091 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.776009    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 19:39:28.778435 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.778398    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 19:39:28.781956 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.781819    2574 manager.go:216] Machine: {Timestamp:2026-09-16 19:39:28.780652649 +0000 UTC m=+0.337997747 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099667 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2912b1053cbf424e78f67d8303ae70 SystemUUID:ec2912b1-053c-bf42-4e78-f67d8303ae70 BootID:03b49a45-150e-4ed1-beb6-2f273572193b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:41:77:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:41:77:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:78:fa:55:36:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 19:39:28.781956 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.781950    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 19:39:28.782085 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.782045    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 19:39:28.785435 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.785391    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 19:39:28.785606 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.785437    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 19:39:28.786119 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.786108    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 19:39:28.786151 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.786121    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 19:39:28.786151 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.786130    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 19:39:28.786696 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.786685    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 19:39:28.787424 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.787397    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 19:39:28.791349 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.791334    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 19:39:28.791495 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.791484    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 19:39:28.791534 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.791515    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 19:39:28.791534 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.791526    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 19:39:28.791586 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.791545    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 19:39:28.791586 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.791556    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 16 19:39:28.791586 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.791565    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 19:39:28.791959 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.791945    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 19:39:28.792027 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.791970    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 19:39:28.794091 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.794075    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 19:39:28.795894 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.795874    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 19:39:28.795994 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.795908    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 19:39:28.797622 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797610    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 19:39:28.797670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797627    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 19:39:28.797670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797633    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 19:39:28.797670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797640    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 19:39:28.797670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797649    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 19:39:28.797670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797655    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 19:39:28.797670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797668    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 19:39:28.797823 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797673    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 19:39:28.797823 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797684    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 19:39:28.797823 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797690    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 19:39:28.797823 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797703    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 19:39:28.797823 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.797708    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 19:39:28.799261 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.799250    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 19:39:28.799261 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.799259    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 19:39:28.803077 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.803063    2574 server.go:1268] "Started kubelet"
Sep 16 19:39:28.805951 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.805891    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 19:39:28.806127 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.803174    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 19:39:28.806249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.806156    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 19:39:28.806760 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.806654    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-26.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 19:39:28.806819 ip-10-0-141-26 systemd[1]: Started Kubernetes Kubelet.
Sep 16 19:39:28.808631 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:28.807349    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 19:39:28.808631 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:28.807531    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 19:39:28.808631 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.808281    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 19:39:28.810000 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.809983    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 19:39:28.810690 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.810668    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8k82b"
Sep 16 19:39:28.811710 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:28.811670    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 19:39:28.813430 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.813413    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 19:39:28.813509 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.813432    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 19:39:28.813688 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:28.813534    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:28.814062 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814043    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 19:39:28.814062 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814064    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 19:39:28.814188 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814168    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 19:39:28.814188 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814176    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 19:39:28.814292 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814194    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 19:39:28.814292 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814241    2574 factory.go:55] Registering systemd factory
Sep 16 19:39:28.814292 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814257    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 16 19:39:28.814514 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814486    2574 factory.go:153] Registering CRI-O factory
Sep 16 19:39:28.814514 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814506    2574 factory.go:223] Registration of the crio container factory successfully
Sep 16 19:39:28.814649 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814561    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 19:39:28.814649 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814580    2574 factory.go:103] Registering Raw factory
Sep 16 19:39:28.814649 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814595    2574 manager.go:1196] Started watching for new ooms in manager
Sep 16 19:39:28.814994 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.814982    2574 manager.go:349] Starting recovery of all containers
Sep 16 19:39:28.816548 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:28.816396    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 19:39:28.816652 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:28.815535    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-26.ec2.internal.18d5e46984f5b036  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-26.ec2.internal,UID:ip-10-0-141-26.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-26.ec2.internal,},FirstTimestamp:2026-09-16 19:39:28.803033142 +0000 UTC m=+0.360378219,LastTimestamp:2026-09-16 19:39:28.803033142 +0000 UTC m=+0.360378219,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-26.ec2.internal,}"
Sep 16 19:39:28.816739 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:28.816681    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 19:39:28.821514 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.821489    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8k82b"
Sep 16 19:39:28.823880 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.823862    2574 manager.go:354] Recovery completed
Sep 16 19:39:28.828352 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.828336    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:28.830152 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.830131    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:28.830269 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.830168    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:28.830269 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.830183    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:28.830788 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.830772    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 19:39:28.830859 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.830790    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 19:39:28.830859 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.830811    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 19:39:28.830859 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.830777    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:28.831518 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.831503    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:28.831577 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.831536    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:28.831577 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.831554    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:28.832634 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:28.832552    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-26.ec2.internal.18d5e46986937986  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-26.ec2.internal,UID:ip-10-0-141-26.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-26.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-26.ec2.internal,},FirstTimestamp:2026-09-16 19:39:28.830151046 +0000 UTC m=+0.387496127,LastTimestamp:2026-09-16 19:39:28.830151046 +0000 UTC m=+0.387496127,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-26.ec2.internal,}"
Sep 16 19:39:28.832973 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.832956    2574 policy_none.go:49] "None policy: Start"
Sep 16 19:39:28.832973 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.832975    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 19:39:28.833068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.832985    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 19:39:28.914361 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:28.914319    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:28.980434 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:28.980336    2574 policy_none.go:47] "Start"
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.012106    2574 manager.go:337] "Starting Device Plugin manager"
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.012151    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.012161    2574 server.go:87] "Starting device plugin registration server"
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.012543    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.012556    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.012664    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.012750    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.012761    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.013330    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 19:39:29.015036 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.013388    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.022917 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.022895    2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-26.ec2.internal\" not found" node="ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.080686 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.080641    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 19:39:29.081875 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.081855    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 19:39:29.081999 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.081911    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 19:39:29.081999 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.081947    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 19:39:29.082089 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.081997    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 19:39:29.085042 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.085017    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:29.112847 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.112812    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:29.113886 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.113867    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:29.113992 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.113902    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:29.113992 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.113915    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:29.113992 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.113945    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.120308 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.120289    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.120415 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.120315    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-26.ec2.internal\": node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.140969 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.140943    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.182795 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.182753    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal"]
Sep 16 19:39:29.182889 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.182866    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:29.184412 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.184385    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:29.184481 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.184436    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:29.184481 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.184449    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:29.186689 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.186676    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:29.186868 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.186853    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.186918 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.186885    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:29.187519 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.187495    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:29.187613 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.187528    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:29.187613 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.187533    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:29.187613 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.187542    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:29.187613 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.187576    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:29.187613 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.187589    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:29.189776 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.189762    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.189819 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.189788    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 19:39:29.190594 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.190577    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 19:39:29.190683 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.190609    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 19:39:29.190683 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.190618    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeHasSufficientPID"
Sep 16 19:39:29.207036 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.207009    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-26.ec2.internal\" not found" node="ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.210579 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.210559    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-26.ec2.internal\" not found" node="ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.215744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.215724    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f29c3797048cea5b6127d9fe62b897c7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal\" (UID: \"f29c3797048cea5b6127d9fe62b897c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.215811 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.215758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f29c3797048cea5b6127d9fe62b897c7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal\" (UID: \"f29c3797048cea5b6127d9fe62b897c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.215811 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.215781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d3f4231defbbc87c04f877cedab58518-config\") pod \"kube-apiserver-proxy-ip-10-0-141-26.ec2.internal\" (UID: \"d3f4231defbbc87c04f877cedab58518\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.241726 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.241651    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.315974 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.315939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f29c3797048cea5b6127d9fe62b897c7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal\" (UID: \"f29c3797048cea5b6127d9fe62b897c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.315974 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.315976    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f29c3797048cea5b6127d9fe62b897c7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal\" (UID: \"f29c3797048cea5b6127d9fe62b897c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.316197 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.315999    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d3f4231defbbc87c04f877cedab58518-config\") pod \"kube-apiserver-proxy-ip-10-0-141-26.ec2.internal\" (UID: \"d3f4231defbbc87c04f877cedab58518\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.316197 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.316064    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d3f4231defbbc87c04f877cedab58518-config\") pod \"kube-apiserver-proxy-ip-10-0-141-26.ec2.internal\" (UID: \"d3f4231defbbc87c04f877cedab58518\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.316197 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.316083    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f29c3797048cea5b6127d9fe62b897c7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal\" (UID: \"f29c3797048cea5b6127d9fe62b897c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.316197 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.316091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f29c3797048cea5b6127d9fe62b897c7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal\" (UID: \"f29c3797048cea5b6127d9fe62b897c7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.342389 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.342360    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.444068 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.444019    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.513164 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.513085    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.513164 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.513125    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:29.544822 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.544785    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.647130 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.647090    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.745861 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.745824    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 19:39:29.746304 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.746027    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 19:39:29.748054 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.748029    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.826442 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.826330    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 19:34:29 +0000 UTC" deadline="2028-04-11 06:44:35.473408731 +0000 UTC"
Sep 16 19:39:29.826442 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.826363    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13739h5m5.647049892s"
Sep 16 19:39:29.848481 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.848447    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:29.916767 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:29.916735    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:29.949298 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:29.949231    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:30.007297 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.007264    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:30.050470 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.050436    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:30.150984 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.150955    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:30.252161 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.252128    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:30.297271 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:30.297232    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f4231defbbc87c04f877cedab58518.slice/crio-6f8073acfa1bbed5ce7e47593ea664f069e1b5c88af31f9973f2c03436810896 WatchSource:0}: Error finding container 6f8073acfa1bbed5ce7e47593ea664f069e1b5c88af31f9973f2c03436810896: Status 404 returned error can't find the container with id 6f8073acfa1bbed5ce7e47593ea664f069e1b5c88af31f9973f2c03436810896
Sep 16 19:39:30.297602 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:30.297578    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29c3797048cea5b6127d9fe62b897c7.slice/crio-a26173ca94778a9f7d7c396b1d3747adae1ecd2be676f4c58679b1e43e69bcb1 WatchSource:0}: Error finding container a26173ca94778a9f7d7c396b1d3747adae1ecd2be676f4c58679b1e43e69bcb1: Status 404 returned error can't find the container with id a26173ca94778a9f7d7c396b1d3747adae1ecd2be676f4c58679b1e43e69bcb1
Sep 16 19:39:30.304736 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.304707    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:39:30.354074 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.354009    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-26.ec2.internal\" not found"
Sep 16 19:39:30.385798 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.385739    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:30.415087 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.415053    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:30.426984 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.426957    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 19:39:30.428268 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.428251    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal"
Sep 16 19:39:30.437890 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.437858    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 19:39:30.793016 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.792925    2574 apiserver.go:52] "Watching apiserver"
Sep 16 19:39:30.801506 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.801478    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 19:39:30.801923 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.801899    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-lcb2q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg","openshift-cluster-node-tuning-operator/tuned-ns6tm","openshift-multus/network-metrics-daemon-9fgnw","openshift-network-diagnostics/network-check-target-vmphk","openshift-network-operator/iptables-alerter-2qms4","openshift-ovn-kubernetes/ovnkube-node-ksztl","kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal","openshift-dns/node-resolver-n6mm2","openshift-image-registry/node-ca-jdb2d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal","openshift-multus/multus-additional-cni-plugins-kxm92","openshift-multus/multus-bwh92","kube-system/global-pull-secret-syncer-jp5lb"]
Sep 16 19:39:30.804827 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.804800    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.807479 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.807275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.807479 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.807358    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 19:39:30.807657 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.807499    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 19:39:30.807854 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.807831    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 19:39:30.808029 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.808007    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 19:39:30.808148 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.808088    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wlxtx\""
Sep 16 19:39:30.809046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.808812    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 19:39:30.809046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.808828    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 19:39:30.809735 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.809711    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 19:39:30.810012 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.809992    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.810138 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.810121    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 19:39:30.810548 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.810526    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 19:39:30.810772 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.810754    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 19:39:30.810864 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.810790    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kdbbc\""
Sep 16 19:39:30.812281 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.812256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:30.812379 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.812332    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:30.812661 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.812637    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:39:30.812753 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.812696    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 19:39:30.812753 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.812709    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q7qnw\""
Sep 16 19:39:30.814338 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.814316    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 19:39:30.814875 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.814855    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:30.814970 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.814926    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:30.818736 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.817793    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:30.820715 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.820693    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-65kkm\""
Sep 16 19:39:30.821083 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.821062    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:39:30.821083 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.821064    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 19:39:30.821213 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.821201    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 19:39:30.822403 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.822382    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:30.824636 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.824610    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 19:39:30.824770 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.824748    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wp6dc\""
Sep 16 19:39:30.824843 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.824610    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 19:39:30.825157 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825133    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94124ceb-b15b-41c7-afbf-926ef8e70639-env-overrides\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.825272 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzhz\" (UniqueName: \"kubernetes.io/projected/94124ceb-b15b-41c7-afbf-926ef8e70639-kube-api-access-xzzhz\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.825272 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825200    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-cni-dir\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.825272 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-run-netns\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.825467 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825302    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:30.825467 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825368    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-run-ovn-kubernetes\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.825467 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825418    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-var-lib-kubelet\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.825467 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-run-multus-certs\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.825749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-sysctl-conf\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.825749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825571    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-sys\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.825749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825599    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-var-lib-kubelet\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.825749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-run-ovn\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.825749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-node-log\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.825749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825691    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94124ceb-b15b-41c7-afbf-926ef8e70639-ovnkube-config\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.825749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825714    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-socket-dir-parent\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.825749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-run-k8s-cni-cncf-io\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825766    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-daemon-config\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7m5w\" (UniqueName: \"kubernetes.io/projected/82fad23c-10de-49aa-b40d-0d3b45360cbb-kube-api-access-p7m5w\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825813    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-system-cni-dir\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825850    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-var-lib-cni-bin\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-modprobe-d\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825904    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-sysconfig\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-kubernetes\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.825974    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-host\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0388b16d-8105-4f63-a1b2-3fb987546771-etc-tuned\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgfs\" (UniqueName: \"kubernetes.io/projected/0388b16d-8105-4f63-a1b2-3fb987546771-kube-api-access-8pgfs\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826068    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-slash\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-etc-openvswitch\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.826158 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-sysctl-d\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826167    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-lib-modules\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826201    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbm4s\" (UniqueName: \"kubernetes.io/projected/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-kube-api-access-nbm4s\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826230    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-run-netns\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-run-openvswitch\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826276    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-cni-netd\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826298    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94124ceb-b15b-41c7-afbf-926ef8e70639-ovn-node-metrics-cert\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826325    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82fad23c-10de-49aa-b40d-0d3b45360cbb-cni-binary-copy\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826364    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-systemd\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-kubelet\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-run-systemd\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826478    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-var-lib-openvswitch\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826504    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-cnibin\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826526    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-etc-kubernetes\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-run\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.826744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.827627 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826596    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94124ceb-b15b-41c7-afbf-926ef8e70639-ovnkube-script-lib\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.827627 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-var-lib-cni-multus\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.827627 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826643    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-hostroot\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.827627 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-conf-dir\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.827627 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0388b16d-8105-4f63-a1b2-3fb987546771-tmp\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.827627 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-systemd-units\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.827627 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826721    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-log-socket\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.827627 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.826744    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-cni-bin\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.828364 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.828340    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:30.828806 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.828784    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 19:39:30.830926 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.830906    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-phw22\""
Sep 16 19:39:30.831027 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.830950    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 19:39:30.831027 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.830985    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 19:39:30.831377 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.831357    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:30.833793 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.833771    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 19:39:30.834110 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.834093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:30.834180 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.834156    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 19:39:30.834577 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.834558    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wdgn8\""
Sep 16 19:39:30.834744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.834720    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 19:39:30.836315 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.836293    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 19:39:30.837224 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.836708    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 19:39:30.837224 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.836779    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 19:39:30.837224 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.836830    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:30.837224 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.836713    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j8thk\""
Sep 16 19:39:30.837224 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.836907    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:30.839382 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.839363    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:30.841950 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.841752    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rdb8s\""
Sep 16 19:39:30.841950 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.841759    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 19:39:30.842123 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.841962    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 19:39:30.849670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.849600    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sdb6l"
Sep 16 19:39:30.856423 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.856398    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sdb6l"
Sep 16 19:39:30.915993 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.915956    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 19:39:30.927875 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.927835    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqr9\" (UniqueName: \"kubernetes.io/projected/d577c619-6162-4417-8e78-f3bff26da8b7-kube-api-access-rdqr9\") pod \"node-resolver-n6mm2\" (UID: \"d577c619-6162-4417-8e78-f3bff26da8b7\") " pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:30.928046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.927893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.928046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.927922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-var-lib-cni-multus\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.928046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.927949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-hostroot\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.928046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.927975    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0388b16d-8105-4f63-a1b2-3fb987546771-tmp\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.928046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.927999    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f185472-cb62-407e-8507-df45a42c37e2-serviceca\") pod \"node-ca-jdb2d\" (UID: \"8f185472-cb62-407e-8507-df45a42c37e2\") " pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:30.928046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928009    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.928046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928025    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ecc37a3c-9ad5-48c3-858e-31e943c27f92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928051    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-registration-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-systemd-units\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-cni-dir\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928121    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-run-netns\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-var-lib-kubelet\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-run-multus-certs\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928240    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-run-ovn-kubernetes\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-var-lib-kubelet\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zclxk\" (UniqueName: \"kubernetes.io/projected/ecc37a3c-9ad5-48c3-858e-31e943c27f92-kube-api-access-zclxk\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-node-log\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928350    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-socket-dir-parent\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928383    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-daemon-config\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928406    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7m5w\" (UniqueName: \"kubernetes.io/projected/82fad23c-10de-49aa-b40d-0d3b45360cbb-kube-api-access-p7m5w\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-kubernetes\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srnbr\" (UniqueName: \"kubernetes.io/projected/8f185472-cb62-407e-8507-df45a42c37e2-kube-api-access-srnbr\") pod \"node-ca-jdb2d\" (UID: \"8f185472-cb62-407e-8507-df45a42c37e2\") " pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:30.928529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-var-lib-cni-bin\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-host\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0388b16d-8105-4f63-a1b2-3fb987546771-etc-tuned\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c11e1d7a-2544-4453-9d6c-6f306de66d66-agent-certs\") pod \"konnectivity-agent-lcb2q\" (UID: \"c11e1d7a-2544-4453-9d6c-6f306de66d66\") " pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f185472-cb62-407e-8507-df45a42c37e2-host\") pod \"node-ca-jdb2d\" (UID: \"8f185472-cb62-407e-8507-df45a42c37e2\") " pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ecc37a3c-9ad5-48c3-858e-31e943c27f92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-socket-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-sysctl-d\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbm4s\" (UniqueName: \"kubernetes.io/projected/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-kube-api-access-nbm4s\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928681    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928717    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c11e1d7a-2544-4453-9d6c-6f306de66d66-konnectivity-ca\") pod \"konnectivity-agent-lcb2q\" (UID: \"c11e1d7a-2544-4453-9d6c-6f306de66d66\") " pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecc37a3c-9ad5-48c3-858e-31e943c27f92-system-cni-dir\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928824    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-run-netns\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-run-openvswitch\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94124ceb-b15b-41c7-afbf-926ef8e70639-ovn-node-metrics-cert\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928907    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82fad23c-10de-49aa-b40d-0d3b45360cbb-cni-binary-copy\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.929303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-systemd\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928957    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928955    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.928981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d577c619-6162-4417-8e78-f3bff26da8b7-hosts-file\") pod \"node-resolver-n6mm2\" (UID: \"d577c619-6162-4417-8e78-f3bff26da8b7\") " pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-kubelet\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-etc-kubernetes\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929054    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a23cd2b-f203-4b66-af13-18bfd2cbdda2-iptables-alerter-script\") pod \"iptables-alerter-2qms4\" (UID: \"4a23cd2b-f203-4b66-af13-18bfd2cbdda2\") " pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929087    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-sys-fs\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929117    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94124ceb-b15b-41c7-afbf-926ef8e70639-ovnkube-script-lib\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-conf-dir\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-kubelet-config\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-dbus\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-log-socket\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-cni-bin\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929323    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-var-lib-cni-multus\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94124ceb-b15b-41c7-afbf-926ef8e70639-env-overrides\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929369    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-hostroot\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929471    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-cni-bin\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-kubelet\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929512    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-run-openvswitch\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929530    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-sysctl-d\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929511    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-conf-dir\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929546    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-systemd\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929609    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-run-ovn-kubernetes\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929653    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-run-netns\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929687    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-var-lib-kubelet\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929681    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-log-socket\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929738    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-socket-dir-parent\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929788    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-node-log\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929758    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-cni-dir\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929800    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-kubernetes\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929843    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-host\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929854    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-var-lib-cni-bin\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929844    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-run-netns\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929909    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-var-lib-kubelet\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.930996 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.929927    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929937    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzhz\" (UniqueName: \"kubernetes.io/projected/94124ceb-b15b-41c7-afbf-926ef8e70639-kube-api-access-xzzhz\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.929967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-sysctl-conf\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.930010    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs podName:4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:31.429972848 +0000 UTC m=+2.987317915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs") pod "network-metrics-daemon-9fgnw" (UID: "4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930082    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-sysctl-conf\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930122    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-systemd-units\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82fad23c-10de-49aa-b40d-0d3b45360cbb-cni-binary-copy\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-sys\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ecc37a3c-9ad5-48c3-858e-31e943c27f92-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ecc37a3c-9ad5-48c3-858e-31e943c27f92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930389    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-etc-kubernetes\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2mm\" (UniqueName: \"kubernetes.io/projected/4a23cd2b-f203-4b66-af13-18bfd2cbdda2-kube-api-access-df2mm\") pod \"iptables-alerter-2qms4\" (UID: \"4a23cd2b-f203-4b66-af13-18bfd2cbdda2\") " pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-run-ovn\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/82fad23c-10de-49aa-b40d-0d3b45360cbb-multus-daemon-config\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94124ceb-b15b-41c7-afbf-926ef8e70639-ovnkube-config\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930583    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-run-k8s-cni-cncf-io\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930642    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-run-ovn\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.931949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-run-multus-certs\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930707    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-sys\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930749    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-host-run-k8s-cni-cncf-io\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930801    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-modprobe-d\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577c619-6162-4417-8e78-f3bff26da8b7-tmp-dir\") pod \"node-resolver-n6mm2\" (UID: \"d577c619-6162-4417-8e78-f3bff26da8b7\") " pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930844    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94124ceb-b15b-41c7-afbf-926ef8e70639-env-overrides\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930884    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94124ceb-b15b-41c7-afbf-926ef8e70639-ovnkube-script-lib\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-system-cni-dir\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930925    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-modprobe-d\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930937    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-sysconfig\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930983    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-system-cni-dir\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.930989    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-etc-sysconfig\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgfs\" (UniqueName: \"kubernetes.io/projected/0388b16d-8105-4f63-a1b2-3fb987546771-kube-api-access-8pgfs\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a23cd2b-f203-4b66-af13-18bfd2cbdda2-host-slash\") pod \"iptables-alerter-2qms4\" (UID: \"4a23cd2b-f203-4b66-af13-18bfd2cbdda2\") " pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-device-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931167    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94124ceb-b15b-41c7-afbf-926ef8e70639-ovnkube-config\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-slash\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-etc-openvswitch\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.932874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931249    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-slash\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931289    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-lib-modules\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931304    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-etc-openvswitch\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-cni-netd\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ecc37a3c-9ad5-48c3-858e-31e943c27f92-cnibin\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931423    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-lib-modules\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931457    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-host-cni-netd\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931503    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzt9q\" (UniqueName: \"kubernetes.io/projected/4671352d-1075-4850-a25b-4df0189be941-kube-api-access-fzt9q\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931544    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-run-systemd\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-var-lib-openvswitch\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-cnibin\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931663    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-run-systemd\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931706    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82fad23c-10de-49aa-b40d-0d3b45360cbb-cnibin\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931706    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94124ceb-b15b-41c7-afbf-926ef8e70639-var-lib-openvswitch\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-run\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.933546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.931839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0388b16d-8105-4f63-a1b2-3fb987546771-run\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.935236 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.934915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94124ceb-b15b-41c7-afbf-926ef8e70639-ovn-node-metrics-cert\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.935497 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.935428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0388b16d-8105-4f63-a1b2-3fb987546771-tmp\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.936394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.936344    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0388b16d-8105-4f63-a1b2-3fb987546771-etc-tuned\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.941718 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.941694    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:30.941849 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.941723    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:30.941849 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.941737    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zf6gj for pod openshift-network-diagnostics/network-check-target-vmphk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:30.941849 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:30.941805    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj podName:fdccdb4c-02fe-4d84-a20a-dc9d12b896c8 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:31.441777971 +0000 UTC m=+2.999123054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zf6gj" (UniqueName: "kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj") pod "network-check-target-vmphk" (UID: "fdccdb4c-02fe-4d84-a20a-dc9d12b896c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:30.942572 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.942420    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgfs\" (UniqueName: \"kubernetes.io/projected/0388b16d-8105-4f63-a1b2-3fb987546771-kube-api-access-8pgfs\") pod \"tuned-ns6tm\" (UID: \"0388b16d-8105-4f63-a1b2-3fb987546771\") " pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:30.944007 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.943967    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbm4s\" (UniqueName: \"kubernetes.io/projected/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-kube-api-access-nbm4s\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:30.947028 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.947005    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzhz\" (UniqueName: \"kubernetes.io/projected/94124ceb-b15b-41c7-afbf-926ef8e70639-kube-api-access-xzzhz\") pod \"ovnkube-node-ksztl\" (UID: \"94124ceb-b15b-41c7-afbf-926ef8e70639\") " pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:30.947879 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:30.947855    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7m5w\" (UniqueName: \"kubernetes.io/projected/82fad23c-10de-49aa-b40d-0d3b45360cbb-kube-api-access-p7m5w\") pod \"multus-bwh92\" (UID: \"82fad23c-10de-49aa-b40d-0d3b45360cbb\") " pod="openshift-multus/multus-bwh92"
Sep 16 19:39:31.032811 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.032776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c11e1d7a-2544-4453-9d6c-6f306de66d66-agent-certs\") pod \"konnectivity-agent-lcb2q\" (UID: \"c11e1d7a-2544-4453-9d6c-6f306de66d66\") " pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:31.032811 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.032816    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f185472-cb62-407e-8507-df45a42c37e2-host\") pod \"node-ca-jdb2d\" (UID: \"8f185472-cb62-407e-8507-df45a42c37e2\") " pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:31.033071 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.032848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ecc37a3c-9ad5-48c3-858e-31e943c27f92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.033071 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.032879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-socket-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.033071 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.032905    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f185472-cb62-407e-8507-df45a42c37e2-host\") pod \"node-ca-jdb2d\" (UID: \"8f185472-cb62-407e-8507-df45a42c37e2\") " pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:31.033071 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.032905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:31.033071 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.032959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c11e1d7a-2544-4453-9d6c-6f306de66d66-konnectivity-ca\") pod \"konnectivity-agent-lcb2q\" (UID: \"c11e1d7a-2544-4453-9d6c-6f306de66d66\") " pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:31.033071 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.033005    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:31.033071 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecc37a3c-9ad5-48c3-858e-31e943c27f92-system-cni-dir\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.033087    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret podName:c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa nodeName:}" failed. No retries permitted until 2026-09-16 19:39:31.533064516 +0000 UTC m=+3.090409624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret") pod "global-pull-secret-syncer-jp5lb" (UID: "c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033120    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d577c619-6162-4417-8e78-f3bff26da8b7-hosts-file\") pod \"node-resolver-n6mm2\" (UID: \"d577c619-6162-4417-8e78-f3bff26da8b7\") " pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033203    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a23cd2b-f203-4b66-af13-18bfd2cbdda2-iptables-alerter-script\") pod \"iptables-alerter-2qms4\" (UID: \"4a23cd2b-f203-4b66-af13-18bfd2cbdda2\") " pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-socket-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-sys-fs\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033256    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecc37a3c-9ad5-48c3-858e-31e943c27f92-system-cni-dir\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-kubelet-config\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033300    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-sys-fs\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033316    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-dbus\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033344    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033360    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d577c619-6162-4417-8e78-f3bff26da8b7-hosts-file\") pod \"node-resolver-n6mm2\" (UID: \"d577c619-6162-4417-8e78-f3bff26da8b7\") " pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033375    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ecc37a3c-9ad5-48c3-858e-31e943c27f92-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ecc37a3c-9ad5-48c3-858e-31e943c27f92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.033439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033429    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2mm\" (UniqueName: \"kubernetes.io/projected/4a23cd2b-f203-4b66-af13-18bfd2cbdda2-kube-api-access-df2mm\") pod \"iptables-alerter-2qms4\" (UID: \"4a23cd2b-f203-4b66-af13-18bfd2cbdda2\") " pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033507    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577c619-6162-4417-8e78-f3bff26da8b7-tmp-dir\") pod \"node-resolver-n6mm2\" (UID: \"d577c619-6162-4417-8e78-f3bff26da8b7\") " pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033537    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a23cd2b-f203-4b66-af13-18bfd2cbdda2-host-slash\") pod \"iptables-alerter-2qms4\" (UID: \"4a23cd2b-f203-4b66-af13-18bfd2cbdda2\") " pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033556    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ecc37a3c-9ad5-48c3-858e-31e943c27f92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033564    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-device-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-device-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033440    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-dbus\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ecc37a3c-9ad5-48c3-858e-31e943c27f92-cnibin\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033675    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a23cd2b-f203-4b66-af13-18bfd2cbdda2-host-slash\") pod \"iptables-alerter-2qms4\" (UID: \"4a23cd2b-f203-4b66-af13-18bfd2cbdda2\") " pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033689    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ecc37a3c-9ad5-48c3-858e-31e943c27f92-cnibin\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033798    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-kubelet-config\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033853    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033864    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ecc37a3c-9ad5-48c3-858e-31e943c27f92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033905    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a23cd2b-f203-4b66-af13-18bfd2cbdda2-iptables-alerter-script\") pod \"iptables-alerter-2qms4\" (UID: \"4a23cd2b-f203-4b66-af13-18bfd2cbdda2\") " pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c11e1d7a-2544-4453-9d6c-6f306de66d66-konnectivity-ca\") pod \"konnectivity-agent-lcb2q\" (UID: \"c11e1d7a-2544-4453-9d6c-6f306de66d66\") " pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033954    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzt9q\" (UniqueName: \"kubernetes.io/projected/4671352d-1075-4850-a25b-4df0189be941-kube-api-access-fzt9q\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.034247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.033988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqr9\" (UniqueName: \"kubernetes.io/projected/d577c619-6162-4417-8e78-f3bff26da8b7-kube-api-access-rdqr9\") pod \"node-resolver-n6mm2\" (UID: \"d577c619-6162-4417-8e78-f3bff26da8b7\") " pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034006    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577c619-6162-4417-8e78-f3bff26da8b7-tmp-dir\") pod \"node-resolver-n6mm2\" (UID: \"d577c619-6162-4417-8e78-f3bff26da8b7\") " pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034020    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f185472-cb62-407e-8507-df45a42c37e2-serviceca\") pod \"node-ca-jdb2d\" (UID: \"8f185472-cb62-407e-8507-df45a42c37e2\") " pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ecc37a3c-9ad5-48c3-858e-31e943c27f92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-registration-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034135    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ecc37a3c-9ad5-48c3-858e-31e943c27f92-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034153    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4671352d-1075-4850-a25b-4df0189be941-registration-dir\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zclxk\" (UniqueName: \"kubernetes.io/projected/ecc37a3c-9ad5-48c3-858e-31e943c27f92-kube-api-access-zclxk\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srnbr\" (UniqueName: \"kubernetes.io/projected/8f185472-cb62-407e-8507-df45a42c37e2-kube-api-access-srnbr\") pod \"node-ca-jdb2d\" (UID: \"8f185472-cb62-407e-8507-df45a42c37e2\") " pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034547    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f185472-cb62-407e-8507-df45a42c37e2-serviceca\") pod \"node-ca-jdb2d\" (UID: \"8f185472-cb62-407e-8507-df45a42c37e2\") " pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:31.035134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.034648    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ecc37a3c-9ad5-48c3-858e-31e943c27f92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.036130 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.036102    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c11e1d7a-2544-4453-9d6c-6f306de66d66-agent-certs\") pod \"konnectivity-agent-lcb2q\" (UID: \"c11e1d7a-2544-4453-9d6c-6f306de66d66\") " pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:31.055546 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.055476    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 19:39:31.056776 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.056749    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zclxk\" (UniqueName: \"kubernetes.io/projected/ecc37a3c-9ad5-48c3-858e-31e943c27f92-kube-api-access-zclxk\") pod \"multus-additional-cni-plugins-kxm92\" (UID: \"ecc37a3c-9ad5-48c3-858e-31e943c27f92\") " pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.061763 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.061735    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srnbr\" (UniqueName: \"kubernetes.io/projected/8f185472-cb62-407e-8507-df45a42c37e2-kube-api-access-srnbr\") pod \"node-ca-jdb2d\" (UID: \"8f185472-cb62-407e-8507-df45a42c37e2\") " pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:31.064049 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.063170    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2mm\" (UniqueName: \"kubernetes.io/projected/4a23cd2b-f203-4b66-af13-18bfd2cbdda2-kube-api-access-df2mm\") pod \"iptables-alerter-2qms4\" (UID: \"4a23cd2b-f203-4b66-af13-18bfd2cbdda2\") " pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:31.065014 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.064966    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqr9\" (UniqueName: \"kubernetes.io/projected/d577c619-6162-4417-8e78-f3bff26da8b7-kube-api-access-rdqr9\") pod \"node-resolver-n6mm2\" (UID: \"d577c619-6162-4417-8e78-f3bff26da8b7\") " pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:31.065627 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.065606    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzt9q\" (UniqueName: \"kubernetes.io/projected/4671352d-1075-4850-a25b-4df0189be941-kube-api-access-fzt9q\") pod \"aws-ebs-csi-driver-node-s4wcg\" (UID: \"4671352d-1075-4850-a25b-4df0189be941\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.093582 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.093520    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal" event={"ID":"f29c3797048cea5b6127d9fe62b897c7","Type":"ContainerStarted","Data":"a26173ca94778a9f7d7c396b1d3747adae1ecd2be676f4c58679b1e43e69bcb1"}
Sep 16 19:39:31.095485 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.095450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal" event={"ID":"d3f4231defbbc87c04f877cedab58518","Type":"ContainerStarted","Data":"6f8073acfa1bbed5ce7e47593ea664f069e1b5c88af31f9973f2c03436810896"}
Sep 16 19:39:31.118999 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.118964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:31.129898 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.129682    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwh92"
Sep 16 19:39:31.131750 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:31.131716    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94124ceb_b15b_41c7_afbf_926ef8e70639.slice/crio-ab9e89d7e467b78613e4757da62aeeef3e54a8488f1e93459fe568c916e5b561 WatchSource:0}: Error finding container ab9e89d7e467b78613e4757da62aeeef3e54a8488f1e93459fe568c916e5b561: Status 404 returned error can't find the container with id ab9e89d7e467b78613e4757da62aeeef3e54a8488f1e93459fe568c916e5b561
Sep 16 19:39:31.140463 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.140223    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ns6tm"
Sep 16 19:39:31.143484 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:31.143421    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fad23c_10de_49aa_b40d_0d3b45360cbb.slice/crio-a885e146946400bb525193d1c96211cc44bca63bac67bc6dfd1043a0c44fadf6 WatchSource:0}: Error finding container a885e146946400bb525193d1c96211cc44bca63bac67bc6dfd1043a0c44fadf6: Status 404 returned error can't find the container with id a885e146946400bb525193d1c96211cc44bca63bac67bc6dfd1043a0c44fadf6
Sep 16 19:39:31.147419 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.147247    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qms4"
Sep 16 19:39:31.157914 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.157892    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n6mm2"
Sep 16 19:39:31.159423 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:31.159392    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a23cd2b_f203_4b66_af13_18bfd2cbdda2.slice/crio-9a0171ab8e0c77e4a44e614fa2762f037b6f15ea5991ec31caf29c43ff138ee1 WatchSource:0}: Error finding container 9a0171ab8e0c77e4a44e614fa2762f037b6f15ea5991ec31caf29c43ff138ee1: Status 404 returned error can't find the container with id 9a0171ab8e0c77e4a44e614fa2762f037b6f15ea5991ec31caf29c43ff138ee1
Sep 16 19:39:31.166374 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.166355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:31.172876 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:31.172755    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd577c619_6162_4417_8e78_f3bff26da8b7.slice/crio-ae2605cf02113e7f010750d5320a4e116390fdedec35fd5c9b742d46c8524551 WatchSource:0}: Error finding container ae2605cf02113e7f010750d5320a4e116390fdedec35fd5c9b742d46c8524551: Status 404 returned error can't find the container with id ae2605cf02113e7f010750d5320a4e116390fdedec35fd5c9b742d46c8524551
Sep 16 19:39:31.175313 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.175287    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg"
Sep 16 19:39:31.179738 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:31.179710    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11e1d7a_2544_4453_9d6c_6f306de66d66.slice/crio-206ff35e7330190c89267272e9867913c0c623ec56f345f3921a0cd1a6f63033 WatchSource:0}: Error finding container 206ff35e7330190c89267272e9867913c0c623ec56f345f3921a0cd1a6f63033: Status 404 returned error can't find the container with id 206ff35e7330190c89267272e9867913c0c623ec56f345f3921a0cd1a6f63033
Sep 16 19:39:31.185019 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.184935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jdb2d"
Sep 16 19:39:31.187519 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:31.187415    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4671352d_1075_4850_a25b_4df0189be941.slice/crio-47282e0a2b3f7e125eea0c39a32e883fa28775ee0ba671dedbbb505104284358 WatchSource:0}: Error finding container 47282e0a2b3f7e125eea0c39a32e883fa28775ee0ba671dedbbb505104284358: Status 404 returned error can't find the container with id 47282e0a2b3f7e125eea0c39a32e883fa28775ee0ba671dedbbb505104284358
Sep 16 19:39:31.196406 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.196209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kxm92"
Sep 16 19:39:31.199658 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:31.199621    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f185472_cb62_407e_8507_df45a42c37e2.slice/crio-3db2c80e1d2bc11cfd4c5458b80caed2cac4f8e4d5c35206d59b3329fa608620 WatchSource:0}: Error finding container 3db2c80e1d2bc11cfd4c5458b80caed2cac4f8e4d5c35206d59b3329fa608620: Status 404 returned error can't find the container with id 3db2c80e1d2bc11cfd4c5458b80caed2cac4f8e4d5c35206d59b3329fa608620
Sep 16 19:39:31.207984 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:39:31.207955    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc37a3c_9ad5_48c3_858e_31e943c27f92.slice/crio-b9e4ee404d07bdef39bf4d7b4b42f220ccfc75acda6e5be78a0611817c0ae8ce WatchSource:0}: Error finding container b9e4ee404d07bdef39bf4d7b4b42f220ccfc75acda6e5be78a0611817c0ae8ce: Status 404 returned error can't find the container with id b9e4ee404d07bdef39bf4d7b4b42f220ccfc75acda6e5be78a0611817c0ae8ce
Sep 16 19:39:31.438178 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.438073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:31.438358 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.438263    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:31.438358 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.438347    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs podName:4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:32.438327446 +0000 UTC m=+3.995672514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs") pod "network-metrics-daemon-9fgnw" (UID: "4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:31.538832 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.538793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:31.538999 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.538870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:31.538999 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.538948    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:31.538999 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.538983    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:31.538999 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.539000    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:31.539168 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.539014    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zf6gj for pod openshift-network-diagnostics/network-check-target-vmphk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:31.539168 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.539022    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret podName:c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa nodeName:}" failed. No retries permitted until 2026-09-16 19:39:32.538999684 +0000 UTC m=+4.096344760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret") pod "global-pull-secret-syncer-jp5lb" (UID: "c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:31.539168 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:31.539061    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj podName:fdccdb4c-02fe-4d84-a20a-dc9d12b896c8 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:32.539047556 +0000 UTC m=+4.096392622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf6gj" (UniqueName: "kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj") pod "network-check-target-vmphk" (UID: "fdccdb4c-02fe-4d84-a20a-dc9d12b896c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:31.860527 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.860373    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:34:31 +0000 UTC" deadline="2028-04-26 23:04:56.550378726 +0000 UTC"
Sep 16 19:39:31.860527 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:31.860508    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14115h25m24.689921511s"
Sep 16 19:39:32.082582 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.082546    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:32.082769 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:32.082686    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:32.099705 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.099662    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lcb2q" event={"ID":"c11e1d7a-2544-4453-9d6c-6f306de66d66","Type":"ContainerStarted","Data":"206ff35e7330190c89267272e9867913c0c623ec56f345f3921a0cd1a6f63033"}
Sep 16 19:39:32.101711 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.101676    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qms4" event={"ID":"4a23cd2b-f203-4b66-af13-18bfd2cbdda2","Type":"ContainerStarted","Data":"9a0171ab8e0c77e4a44e614fa2762f037b6f15ea5991ec31caf29c43ff138ee1"}
Sep 16 19:39:32.103227 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.103194    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ns6tm" event={"ID":"0388b16d-8105-4f63-a1b2-3fb987546771","Type":"ContainerStarted","Data":"dded7fda47ae89ffff32ba5ce7fdbf278ec8e10b23e8f5b4341e14c22798fbfb"}
Sep 16 19:39:32.104682 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.104658    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n6mm2" event={"ID":"d577c619-6162-4417-8e78-f3bff26da8b7","Type":"ContainerStarted","Data":"ae2605cf02113e7f010750d5320a4e116390fdedec35fd5c9b742d46c8524551"}
Sep 16 19:39:32.106563 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.106529    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwh92" event={"ID":"82fad23c-10de-49aa-b40d-0d3b45360cbb","Type":"ContainerStarted","Data":"a885e146946400bb525193d1c96211cc44bca63bac67bc6dfd1043a0c44fadf6"}
Sep 16 19:39:32.108481 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.108444    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" event={"ID":"94124ceb-b15b-41c7-afbf-926ef8e70639","Type":"ContainerStarted","Data":"ab9e89d7e467b78613e4757da62aeeef3e54a8488f1e93459fe568c916e5b561"}
Sep 16 19:39:32.109936 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.109915    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxm92" event={"ID":"ecc37a3c-9ad5-48c3-858e-31e943c27f92","Type":"ContainerStarted","Data":"b9e4ee404d07bdef39bf4d7b4b42f220ccfc75acda6e5be78a0611817c0ae8ce"}
Sep 16 19:39:32.111585 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.111512    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jdb2d" event={"ID":"8f185472-cb62-407e-8507-df45a42c37e2","Type":"ContainerStarted","Data":"3db2c80e1d2bc11cfd4c5458b80caed2cac4f8e4d5c35206d59b3329fa608620"}
Sep 16 19:39:32.113096 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.113071    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg" event={"ID":"4671352d-1075-4850-a25b-4df0189be941","Type":"ContainerStarted","Data":"47282e0a2b3f7e125eea0c39a32e883fa28775ee0ba671dedbbb505104284358"}
Sep 16 19:39:32.448718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.448002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:32.448718 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:32.448190    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:32.448718 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:32.448264    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs podName:4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:34.448241664 +0000 UTC m=+6.005586739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs") pod "network-metrics-daemon-9fgnw" (UID: "4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:32.549399 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.548588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:32.549399 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.548651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:32.549399 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:32.548794    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:32.549399 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:32.548862    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret podName:c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa nodeName:}" failed. No retries permitted until 2026-09-16 19:39:34.548840696 +0000 UTC m=+6.106185767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret") pod "global-pull-secret-syncer-jp5lb" (UID: "c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:32.549399 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:32.548961    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:32.549399 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:32.548975    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:32.549399 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:32.548988    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zf6gj for pod openshift-network-diagnostics/network-check-target-vmphk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:32.549399 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:32.549034    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj podName:fdccdb4c-02fe-4d84-a20a-dc9d12b896c8 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:34.549021411 +0000 UTC m=+6.106366482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf6gj" (UniqueName: "kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj") pod "network-check-target-vmphk" (UID: "fdccdb4c-02fe-4d84-a20a-dc9d12b896c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:32.861035 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.860926    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 19:34:31 +0000 UTC" deadline="2028-03-23 16:03:49.425626308 +0000 UTC"
Sep 16 19:39:32.861035 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:32.860976    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13292h24m16.564655961s"
Sep 16 19:39:33.086071 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:33.085517    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:33.086071 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:33.085673    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:33.086318 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:33.086104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:33.086318 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:33.086202    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:34.083074 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:34.083020    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:34.087517 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:34.083649    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:34.143996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:34.143955    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal" event={"ID":"f29c3797048cea5b6127d9fe62b897c7","Type":"ContainerStarted","Data":"11cbac100dae48156f95ebdff56147e1ad69a3c303831c5b52ab7283ab8ccf71"}
Sep 16 19:39:34.146581 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:34.146552    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal" event={"ID":"d3f4231defbbc87c04f877cedab58518","Type":"ContainerStarted","Data":"fd83fb1be55e1f042e98f36be93966bac9533bd0985bf4c196ff7fc20f6a1904"}
Sep 16 19:39:34.467434 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:34.467335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:34.467653 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:34.467525    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:34.467653 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:34.467599    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs podName:4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:38.467577741 +0000 UTC m=+10.024922843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs") pod "network-metrics-daemon-9fgnw" (UID: "4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:34.568146 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:34.568107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:34.568370 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:34.568172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:34.568370 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:34.568313    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:34.568370 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:34.568349    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:34.568370 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:34.568363    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zf6gj for pod openshift-network-diagnostics/network-check-target-vmphk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:34.568700 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:34.568382    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:34.568700 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:34.568444    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret podName:c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa nodeName:}" failed. No retries permitted until 2026-09-16 19:39:38.568424194 +0000 UTC m=+10.125769267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret") pod "global-pull-secret-syncer-jp5lb" (UID: "c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:34.568700 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:34.568482    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj podName:fdccdb4c-02fe-4d84-a20a-dc9d12b896c8 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:38.568471037 +0000 UTC m=+10.125816111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf6gj" (UniqueName: "kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj") pod "network-check-target-vmphk" (UID: "fdccdb4c-02fe-4d84-a20a-dc9d12b896c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:35.084333 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:35.083554    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:35.084333 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:35.083717    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:35.084333 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:35.084186    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:35.084333 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:35.084287    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:36.082188 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:36.082150    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:36.082365 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:36.082299    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:37.086707 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:37.086050    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:37.086707 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:37.086188    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:37.086707 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:37.086528    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:37.086707 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:37.086620    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:37.155317 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:37.155283    2574 generic.go:356] "Generic (PLEG): container finished" podID="f29c3797048cea5b6127d9fe62b897c7" containerID="11cbac100dae48156f95ebdff56147e1ad69a3c303831c5b52ab7283ab8ccf71" exitCode=0
Sep 16 19:39:37.155509 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:37.155334    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal" event={"ID":"f29c3797048cea5b6127d9fe62b897c7","Type":"ContainerDied","Data":"11cbac100dae48156f95ebdff56147e1ad69a3c303831c5b52ab7283ab8ccf71"}
Sep 16 19:39:37.171872 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:37.171812    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-26.ec2.internal" podStartSLOduration=7.171792146 podStartE2EDuration="7.171792146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:34.178224925 +0000 UTC m=+5.735570019" watchObservedRunningTime="2026-09-16 19:39:37.171792146 +0000 UTC m=+8.729137235"
Sep 16 19:39:38.082910 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:38.082411    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:38.082910 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:38.082549    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:38.502118 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:38.501412    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:38.502118 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:38.501589    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:38.502118 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:38.501660    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs podName:4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:46.501637199 +0000 UTC m=+18.058982280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs") pod "network-metrics-daemon-9fgnw" (UID: "4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:38.601946 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:38.601876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:38.601946 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:38.601944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:38.602185 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:38.602108    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:38.602185 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:38.602121    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:38.602185 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:38.602134    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:38.602185 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:38.602148    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zf6gj for pod openshift-network-diagnostics/network-check-target-vmphk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:38.602464 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:38.602221    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret podName:c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa nodeName:}" failed. No retries permitted until 2026-09-16 19:39:46.602193311 +0000 UTC m=+18.159538390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret") pod "global-pull-secret-syncer-jp5lb" (UID: "c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:38.602464 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:38.602276    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj podName:fdccdb4c-02fe-4d84-a20a-dc9d12b896c8 nodeName:}" failed. No retries permitted until 2026-09-16 19:39:46.602257903 +0000 UTC m=+18.159602971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf6gj" (UniqueName: "kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj") pod "network-check-target-vmphk" (UID: "fdccdb4c-02fe-4d84-a20a-dc9d12b896c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:39.083063 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:39.083029    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:39.083248 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:39.083168    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:39.084560 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:39.084536    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:39.084695 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:39.084619    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:40.082314 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:40.082281    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:40.082802 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:40.082418    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:41.082319 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:41.082279    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:41.082810 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:41.082447    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:41.082810 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:41.082502    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:41.082810 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:41.082614    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:42.082658 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:42.082611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:42.083142 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:42.082754    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:43.082613 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:43.082573    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:43.082613 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:43.082596    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:43.083144 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:43.082725    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:43.083144 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:43.082806    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:44.082825 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:44.082589    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:44.083255 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:44.082909    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:45.082365 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:45.082330    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:45.082565 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:45.082330    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:45.082565 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:45.082494    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:45.082644 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:45.082557    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:46.082664 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:46.082632    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:46.083171 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:46.082745    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:46.562632 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:46.562598    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:46.562794 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:46.562756    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:46.562864 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:46.562848    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs podName:4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:02.562826993 +0000 UTC m=+34.120172059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs") pod "network-metrics-daemon-9fgnw" (UID: "4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:39:46.663761 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:46.663671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:46.663761 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:46.663732    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:46.663960 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:46.663864    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:46.663960 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:46.663907    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:39:46.663960 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:46.663928    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret podName:c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa nodeName:}" failed. No retries permitted until 2026-09-16 19:40:02.663909411 +0000 UTC m=+34.221254496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret") pod "global-pull-secret-syncer-jp5lb" (UID: "c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:39:46.663960 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:46.663930    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:39:46.663960 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:46.663953    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zf6gj for pod openshift-network-diagnostics/network-check-target-vmphk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:46.664164 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:46.663999    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj podName:fdccdb4c-02fe-4d84-a20a-dc9d12b896c8 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:02.663988925 +0000 UTC m=+34.221334012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf6gj" (UniqueName: "kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj") pod "network-check-target-vmphk" (UID: "fdccdb4c-02fe-4d84-a20a-dc9d12b896c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:39:47.083036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:47.083006    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:47.083481 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:47.083006    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:47.083481 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:47.083170    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:47.083481 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:47.083209    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:48.082499 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:48.082452    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:48.082669 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:48.082571    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:49.083356 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:49.083321    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:49.083869 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:49.083432    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:49.083869 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:49.083467    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:49.083869 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:49.083588    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:50.082203 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.082124    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:50.082269 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:50.082242    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:50.183297 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.183127    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal" event={"ID":"f29c3797048cea5b6127d9fe62b897c7","Type":"ContainerStarted","Data":"36ac022b13b2cf745b66f9045c34d2d831b34637db6f18d1a25d622923bc75cf"}
Sep 16 19:39:50.184971 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.184948    2574 generic.go:356] "Generic (PLEG): container finished" podID="ecc37a3c-9ad5-48c3-858e-31e943c27f92" containerID="e70368a3f9edea428b20e3ef98e5e9d89b914214bc87ab17597475d58396b401" exitCode=0
Sep 16 19:39:50.185082 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.185015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxm92" event={"ID":"ecc37a3c-9ad5-48c3-858e-31e943c27f92","Type":"ContainerDied","Data":"e70368a3f9edea428b20e3ef98e5e9d89b914214bc87ab17597475d58396b401"}
Sep 16 19:39:50.186681 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.186653    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jdb2d" event={"ID":"8f185472-cb62-407e-8507-df45a42c37e2","Type":"ContainerStarted","Data":"19afd04bd9581721f473f59edb3d270b177ee47acb76faec9fded33507ad41fa"}
Sep 16 19:39:50.188386 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.188351    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg" event={"ID":"4671352d-1075-4850-a25b-4df0189be941","Type":"ContainerStarted","Data":"3cf64f342d431519f519a090621ab1a992aaf0eb521ea48e74f19b2b801f8142"}
Sep 16 19:39:50.190084 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.190053    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lcb2q" event={"ID":"c11e1d7a-2544-4453-9d6c-6f306de66d66","Type":"ContainerStarted","Data":"edd26df1f7762cefaf2989b479cea0a611e6aa7cae8e4e7f29d15fd5b7ec71fe"}
Sep 16 19:39:50.191877 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.191848    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ns6tm" event={"ID":"0388b16d-8105-4f63-a1b2-3fb987546771","Type":"ContainerStarted","Data":"2c9f59faf5a2f3aec53019e90cd3df5ce67d3ea42e3210184234bff8eeec94aa"}
Sep 16 19:39:50.193494 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.193473    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n6mm2" event={"ID":"d577c619-6162-4417-8e78-f3bff26da8b7","Type":"ContainerStarted","Data":"905a7d7ba7297442827375245e5ecfcb33c3b5ba9842b6464a501ecbfbee9225"}
Sep 16 19:39:50.195115 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.195095    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwh92" event={"ID":"82fad23c-10de-49aa-b40d-0d3b45360cbb","Type":"ContainerStarted","Data":"4fe96b8fe4809f280773bbeaa946724de936b26517245f8b40cc03d9bfb7509e"}
Sep 16 19:39:50.197089 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.197068    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" event={"ID":"94124ceb-b15b-41c7-afbf-926ef8e70639","Type":"ContainerStarted","Data":"aaf8d15ae7f1407cd87dd28c91587854bab23704e77ab4f313ec99c2eb324c1f"}
Sep 16 19:39:50.197181 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.197121    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" event={"ID":"94124ceb-b15b-41c7-afbf-926ef8e70639","Type":"ContainerStarted","Data":"bc5a0839148f2fd1c69b331a296deaa213866821aa38750abd6c642bd82f00f8"}
Sep 16 19:39:50.200320 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.200285    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-26.ec2.internal" podStartSLOduration=20.200273093 podStartE2EDuration="20.200273093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:50.199892984 +0000 UTC m=+21.757238070" watchObservedRunningTime="2026-09-16 19:39:50.200273093 +0000 UTC m=+21.757618179"
Sep 16 19:39:50.218353 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.218273    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n6mm2" podStartSLOduration=6.047113755 podStartE2EDuration="21.218249973s" totalImagesPullingTime="15.171136218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:50.217330292 +0000 UTC m=+21.774675404" watchObservedRunningTime="2026-09-16 19:39:50.218249973 +0000 UTC m=+21.775595061"
Sep 16 19:39:50.232580 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.232521    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jdb2d" podStartSLOduration=2.595905955 podStartE2EDuration="21.232504115s" totalImagesPullingTime="18.63659816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:50.231695258 +0000 UTC m=+21.789040362" watchObservedRunningTime="2026-09-16 19:39:50.232504115 +0000 UTC m=+21.789849203"
Sep 16 19:39:50.256156 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.256111    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ns6tm" podStartSLOduration=2.573087571 podStartE2EDuration="21.256096991s" totalImagesPullingTime="18.68300942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:50.255694702 +0000 UTC m=+21.813039818" watchObservedRunningTime="2026-09-16 19:39:50.256096991 +0000 UTC m=+21.813442079"
Sep 16 19:39:50.275944 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.275889    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lcb2q" podStartSLOduration=2.621441177 podStartE2EDuration="21.27586938s" totalImagesPullingTime="18.654428203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:50.275257704 +0000 UTC m=+21.832602793" watchObservedRunningTime="2026-09-16 19:39:50.27586938 +0000 UTC m=+21.833214468"
Sep 16 19:39:50.314707 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.314609    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bwh92" podStartSLOduration=2.546063041 podStartE2EDuration="21.314589561s" totalImagesPullingTime="18.76852652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:50.31430897 +0000 UTC m=+21.871654057" watchObservedRunningTime="2026-09-16 19:39:50.314589561 +0000 UTC m=+21.871934645"
Sep 16 19:39:50.460476 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.460445    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:50.543245 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.543195    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:50.613313 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.613267    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:50.955863 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:50.955676    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 19:39:51.021477 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.021373    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T19:39:50.955860224Z","UUID":"066f7b41-ff79-411a-88d8-5f10e952f78e","Handler":null,"Name":"","Endpoint":""}
Sep 16 19:39:51.023126 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.023104    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 19:39:51.023249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.023132    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 19:39:51.082727 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.082703    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:51.082860 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:51.082839    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:51.082920 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.082891    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:51.083021 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:51.083000    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:51.202198 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.202106    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg" event={"ID":"4671352d-1075-4850-a25b-4df0189be941","Type":"ContainerStarted","Data":"0bde947b0cd821f5510d8d538c02ba98c24fd94118c344f081754fab8a1bd5f1"}
Sep 16 19:39:51.203950 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.203916    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qms4" event={"ID":"4a23cd2b-f203-4b66-af13-18bfd2cbdda2","Type":"ContainerStarted","Data":"8eb4f253173319464eb1ac620368f2b50f73b889f34e9dab1abeaf7db3e1442a"}
Sep 16 19:39:51.207386 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.207359    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" event={"ID":"94124ceb-b15b-41c7-afbf-926ef8e70639","Type":"ContainerStarted","Data":"93a4b1f632f284d5b41e02544087dfe0916aa5acab9acb5df7bb37ec75efdccc"}
Sep 16 19:39:51.207536 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.207459    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" event={"ID":"94124ceb-b15b-41c7-afbf-926ef8e70639","Type":"ContainerStarted","Data":"264ae90a6a223251bcb7b834c3df8c51de5fe65c9f8c64ba114f9f8aa34181c6"}
Sep 16 19:39:51.207536 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.207515    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" event={"ID":"94124ceb-b15b-41c7-afbf-926ef8e70639","Type":"ContainerStarted","Data":"e15273dc982e67a44b6ea9dd6ff2755fc9803e46561670437de10e49473922c6"}
Sep 16 19:39:51.207639 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.207560    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" event={"ID":"94124ceb-b15b-41c7-afbf-926ef8e70639","Type":"ContainerStarted","Data":"b31dc897987074053d4bbf04d8fef1e68281dec0c4f56328a87b50604f791e8c"}
Sep 16 19:39:51.219359 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.219313    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2qms4" podStartSLOduration=3.536341561 podStartE2EDuration="22.219299612s" totalImagesPullingTime="18.682958051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:51.218901873 +0000 UTC m=+22.776246984" watchObservedRunningTime="2026-09-16 19:39:51.219299612 +0000 UTC m=+22.776644699"
Sep 16 19:39:51.339394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.339346    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:51.439849 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.439813    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:51.540201 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:51.540088    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:52.083243 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:52.083114    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:52.083311 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:52.083252    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:52.197461 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:52.197426    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:52.197994 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:52.197974    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:52.212035 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:52.212008    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg" event={"ID":"4671352d-1075-4850-a25b-4df0189be941","Type":"ContainerStarted","Data":"6f55527e7ced02aa217778b1eedf73bc8feec6b8d6fb2aa1a80b55b82b4bb87f"}
Sep 16 19:39:52.212487 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:52.212363    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:52.212720 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:52.212696    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lcb2q"
Sep 16 19:39:52.236439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:52.236383    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wcg" podStartSLOduration=2.7496633360000002 podStartE2EDuration="23.236368481s" totalImagesPullingTime="20.486705145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:52.236200563 +0000 UTC m=+23.793545680" watchObservedRunningTime="2026-09-16 19:39:52.236368481 +0000 UTC m=+23.793713567"
Sep 16 19:39:52.308752 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:52.308663    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:52.902702 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:52.902641    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:53.004575 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:53.004532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:53.082443 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:53.082396    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:53.082618 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:53.082416    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:53.082618 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:53.082561    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:53.082732 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:53.082625    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:53.219164 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:53.219073    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" event={"ID":"94124ceb-b15b-41c7-afbf-926ef8e70639","Type":"ContainerStarted","Data":"342c2e07cd8e9a0f7bbaf88d9af859afae22268962b1fa82860fd9c95544e76b"}
Sep 16 19:39:54.082892 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:54.082864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:54.083063 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:54.082982    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:54.148960 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:54.148925    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:54.293634 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:54.293597    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:54.393994 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:54.393916    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:55.082522 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.082340    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:55.082647 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.082417    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:55.082647 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:55.082602    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:55.082708 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:55.082692    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:55.205750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.205717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:55.224766 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.224741    2574 generic.go:356] "Generic (PLEG): container finished" podID="ecc37a3c-9ad5-48c3-858e-31e943c27f92" containerID="7e0b4044fbb97e88f3c01bc92c382438e164105c852b9cf2300d14a5d511feb9" exitCode=0
Sep 16 19:39:55.224884 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.224771    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxm92" event={"ID":"ecc37a3c-9ad5-48c3-858e-31e943c27f92","Type":"ContainerDied","Data":"7e0b4044fbb97e88f3c01bc92c382438e164105c852b9cf2300d14a5d511feb9"}
Sep 16 19:39:55.228694 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.228639    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" event={"ID":"94124ceb-b15b-41c7-afbf-926ef8e70639","Type":"ContainerStarted","Data":"ab0311eeef40e75ff6de939100448821dd98e66975fb2b422a76804a41191f23"}
Sep 16 19:39:55.228951 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.228916    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:55.229045 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.228984    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:55.243731 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.243707    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:55.243859 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.243831    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:39:55.290114 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.290083    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:55.322093 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:55.322050    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" podStartSLOduration=7.549475686 podStartE2EDuration="26.322036848s" totalImagesPullingTime="18.772561162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:55.32024081 +0000 UTC m=+26.877585919" watchObservedRunningTime="2026-09-16 19:39:55.322036848 +0000 UTC m=+26.879381935"
Sep 16 19:39:56.082208 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.082175    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:56.082379 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:56.082311    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:56.233655 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.233626    2574 generic.go:356] "Generic (PLEG): container finished" podID="ecc37a3c-9ad5-48c3-858e-31e943c27f92" containerID="16822dff34e509e6444fb67c744411c1309693d74dff76484ab436152e4bda2a" exitCode=0
Sep 16 19:39:56.233789 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.233750    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxm92" event={"ID":"ecc37a3c-9ad5-48c3-858e-31e943c27f92","Type":"ContainerDied","Data":"16822dff34e509e6444fb67c744411c1309693d74dff76484ab436152e4bda2a"}
Sep 16 19:39:56.233789 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.233770    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 19:39:56.639033 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.639001    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jp5lb"]
Sep 16 19:39:56.639503 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.639132    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:56.639503 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:56.639215    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:56.645842 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.645812    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vmphk"]
Sep 16 19:39:56.645959 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.645949    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:56.646053 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:56.646036    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:56.646677 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.646647    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9fgnw"]
Sep 16 19:39:56.646787 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:56.646763    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:56.646877 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:56.646856    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:39:57.160336 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:57.160305    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:57.238287 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:57.238202    2574 generic.go:356] "Generic (PLEG): container finished" podID="ecc37a3c-9ad5-48c3-858e-31e943c27f92" containerID="9e626c5b8e9e8458234f5f48f7ffeec0957c2fd0146921050fdff261887a5df0" exitCode=0
Sep 16 19:39:57.238463 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:57.238290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxm92" event={"ID":"ecc37a3c-9ad5-48c3-858e-31e943c27f92","Type":"ContainerDied","Data":"9e626c5b8e9e8458234f5f48f7ffeec0957c2fd0146921050fdff261887a5df0"}
Sep 16 19:39:57.238463 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:57.238365    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 19:39:58.082527 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:58.082494    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:39:58.082979 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:58.082544    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:39:58.082979 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:39:58.082624    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:39:58.082979 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:58.082621    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:39:58.082979 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:58.082714    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:39:58.082979 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:39:58.082788    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:40:00.082998 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:00.082966    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:40:00.083477 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:00.083094    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:40:00.083477 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:00.083102    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:40:00.083477 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:00.083223    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:40:00.083477 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:00.083258    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:40:00.083477 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:00.083343    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:40:00.618217 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:00.618179    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:40:00.618444 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:00.618424    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 19:40:00.636464 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:00.636423    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" podUID="94124ceb-b15b-41c7-afbf-926ef8e70639" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 16 19:40:00.645326 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:00.645301    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl" podUID="94124ceb-b15b-41c7-afbf-926ef8e70639" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 16 19:40:02.082464 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.082282    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:40:02.082903 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.082286    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:40:02.082903 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.082533    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jp5lb" podUID="c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa"
Sep 16 19:40:02.082903 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.082286    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:40:02.082903 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.082621    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:40:02.082903 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.082692    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vmphk" podUID="fdccdb4c-02fe-4d84-a20a-dc9d12b896c8"
Sep 16 19:40:02.254091 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.254059    2574 generic.go:356] "Generic (PLEG): container finished" podID="ecc37a3c-9ad5-48c3-858e-31e943c27f92" containerID="7ec797b753e5a438f10c2677e19d9b64560c02a070ceeb80bed467aa03325c90" exitCode=0
Sep 16 19:40:02.254253 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.254121    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxm92" event={"ID":"ecc37a3c-9ad5-48c3-858e-31e943c27f92","Type":"ContainerDied","Data":"7ec797b753e5a438f10c2677e19d9b64560c02a070ceeb80bed467aa03325c90"}
Sep 16 19:40:02.583448 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.583393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:40:02.583605 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.583522    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:40:02.583605 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.583571    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs podName:4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:34.583558827 +0000 UTC m=+66.140903891 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs") pod "network-metrics-daemon-9fgnw" (UID: "4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 19:40:02.684486 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.684463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:40:02.684576 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.684504    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:40:02.684626 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.684612    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 19:40:02.684664 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.684626    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 19:40:02.684664 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.684642    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 19:40:02.684664 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.684655    2574 projected.go:196] Error preparing data for projected volume kube-api-access-zf6gj for pod openshift-network-diagnostics/network-check-target-vmphk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:40:02.684751 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.684679    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret podName:c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa nodeName:}" failed. No retries permitted until 2026-09-16 19:40:34.684661525 +0000 UTC m=+66.242006612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret") pod "global-pull-secret-syncer-jp5lb" (UID: "c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa") : object "kube-system"/"original-pull-secret" not registered
Sep 16 19:40:02.684751 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:02.684696    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj podName:fdccdb4c-02fe-4d84-a20a-dc9d12b896c8 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:34.684688258 +0000 UTC m=+66.242033337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf6gj" (UniqueName: "kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj") pod "network-check-target-vmphk" (UID: "fdccdb4c-02fe-4d84-a20a-dc9d12b896c8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 19:40:02.781089 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.781061    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-26.ec2.internal" event="NodeReady"
Sep 16 19:40:02.781249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.781179    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 19:40:02.833970 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.833940    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5qm2f"]
Sep 16 19:40:02.856557 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.856531    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vzhbl"]
Sep 16 19:40:02.856690 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.856656    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:02.859213 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.859188    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 19:40:02.859342 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.859227    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 19:40:02.859461 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.859424    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdb5m\""
Sep 16 19:40:02.876608 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.876582    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5qm2f"]
Sep 16 19:40:02.876709 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.876617    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vzhbl"]
Sep 16 19:40:02.876709 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.876634    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vzhbl"
Sep 16 19:40:02.879316 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.879291    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 19:40:02.879316 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.879294    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 19:40:02.879316 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.879330    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 19:40:02.879548 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.879536    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tf6pp\""
Sep 16 19:40:02.987232 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.987144    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm58l\" (UniqueName: \"kubernetes.io/projected/01b5832d-8fc0-467b-b874-1a22e958fb24-kube-api-access-fm58l\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:02.987232 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.987204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l66d\" (UniqueName: \"kubernetes.io/projected/025c82f2-833e-4969-8602-ef8b0c6c8b64-kube-api-access-5l66d\") pod \"ingress-canary-vzhbl\" (UID: \"025c82f2-833e-4969-8602-ef8b0c6c8b64\") " pod="openshift-ingress-canary/ingress-canary-vzhbl"
Sep 16 19:40:02.987447 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.987267    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01b5832d-8fc0-467b-b874-1a22e958fb24-config-volume\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:02.987447 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.987295    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:02.987447 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.987324    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b5832d-8fc0-467b-b874-1a22e958fb24-tmp-dir\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:02.987447 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.987341    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/025c82f2-833e-4969-8602-ef8b0c6c8b64-cert\") pod \"ingress-canary-vzhbl\" (UID: \"025c82f2-833e-4969-8602-ef8b0c6c8b64\") " pod="openshift-ingress-canary/ingress-canary-vzhbl"
Sep 16 19:40:02.994185 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:02.994156    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:03.057545 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.057506    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:03.088321 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.088292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l66d\" (UniqueName: \"kubernetes.io/projected/025c82f2-833e-4969-8602-ef8b0c6c8b64-kube-api-access-5l66d\") pod \"ingress-canary-vzhbl\" (UID: \"025c82f2-833e-4969-8602-ef8b0c6c8b64\") " pod="openshift-ingress-canary/ingress-canary-vzhbl"
Sep 16 19:40:03.088876 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.088339    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01b5832d-8fc0-467b-b874-1a22e958fb24-config-volume\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:03.088876 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.088363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:03.088876 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.088424    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b5832d-8fc0-467b-b874-1a22e958fb24-tmp-dir\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:03.088876 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.088450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/025c82f2-833e-4969-8602-ef8b0c6c8b64-cert\") pod \"ingress-canary-vzhbl\" (UID: \"025c82f2-833e-4969-8602-ef8b0c6c8b64\") " pod="openshift-ingress-canary/ingress-canary-vzhbl"
Sep 16 19:40:03.088876 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.088490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm58l\" (UniqueName: \"kubernetes.io/projected/01b5832d-8fc0-467b-b874-1a22e958fb24-kube-api-access-fm58l\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:03.088876 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:03.088521    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:03.088876 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:03.088587    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls podName:01b5832d-8fc0-467b-b874-1a22e958fb24 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:03.588571198 +0000 UTC m=+35.145916279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls") pod "dns-default-5qm2f" (UID: "01b5832d-8fc0-467b-b874-1a22e958fb24") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:03.088876 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.088771    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01b5832d-8fc0-467b-b874-1a22e958fb24-tmp-dir\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:03.089178 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.088915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01b5832d-8fc0-467b-b874-1a22e958fb24-config-volume\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:03.092576 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.092559    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/025c82f2-833e-4969-8602-ef8b0c6c8b64-cert\") pod \"ingress-canary-vzhbl\" (UID: \"025c82f2-833e-4969-8602-ef8b0c6c8b64\") " pod="openshift-ingress-canary/ingress-canary-vzhbl"
Sep 16 19:40:03.097562 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.097536    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm58l\" (UniqueName: \"kubernetes.io/projected/01b5832d-8fc0-467b-b874-1a22e958fb24-kube-api-access-fm58l\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:03.097823 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.097804    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l66d\" (UniqueName: \"kubernetes.io/projected/025c82f2-833e-4969-8602-ef8b0c6c8b64-kube-api-access-5l66d\") pod \"ingress-canary-vzhbl\" (UID: \"025c82f2-833e-4969-8602-ef8b0c6c8b64\") " pod="openshift-ingress-canary/ingress-canary-vzhbl"
Sep 16 19:40:03.184523 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.184492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vzhbl"
Sep 16 19:40:03.260093 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.260010    2574 generic.go:356] "Generic (PLEG): container finished" podID="ecc37a3c-9ad5-48c3-858e-31e943c27f92" containerID="c8d0c0f51aa3d4d3c47f2dae5f10fd6a0b00e9c19f501286c78077e30773cb6e" exitCode=0
Sep 16 19:40:03.260093 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.260066    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxm92" event={"ID":"ecc37a3c-9ad5-48c3-858e-31e943c27f92","Type":"ContainerDied","Data":"c8d0c0f51aa3d4d3c47f2dae5f10fd6a0b00e9c19f501286c78077e30773cb6e"}
Sep 16 19:40:03.499268 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.499092    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vzhbl"]
Sep 16 19:40:03.502350 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:40:03.502319    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025c82f2_833e_4969_8602_ef8b0c6c8b64.slice/crio-87afc593605c475ffeaf9258207f7c88322312c5a2bf383e7d536422b3096bda WatchSource:0}: Error finding container 87afc593605c475ffeaf9258207f7c88322312c5a2bf383e7d536422b3096bda: Status 404 returned error can't find the container with id 87afc593605c475ffeaf9258207f7c88322312c5a2bf383e7d536422b3096bda
Sep 16 19:40:03.591667 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:03.591643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:03.591804 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:03.591766    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:03.591842 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:03.591816    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls podName:01b5832d-8fc0-467b-b874-1a22e958fb24 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:04.591803475 +0000 UTC m=+36.149148553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls") pod "dns-default-5qm2f" (UID: "01b5832d-8fc0-467b-b874-1a22e958fb24") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:04.082720 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.082687    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:40:04.082720 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.082723    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:40:04.082990 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.082694    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:40:04.086784 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.086563    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmvpx\""
Sep 16 19:40:04.086784 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.086595    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2djhj\""
Sep 16 19:40:04.086784 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.086635    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 19:40:04.086784 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.086654    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 19:40:04.086784 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.086599    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 19:40:04.086784 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.086781    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 19:40:04.264551 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.264510    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vzhbl" event={"ID":"025c82f2-833e-4969-8602-ef8b0c6c8b64","Type":"ContainerStarted","Data":"87afc593605c475ffeaf9258207f7c88322312c5a2bf383e7d536422b3096bda"}
Sep 16 19:40:04.269301 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.269273    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxm92" event={"ID":"ecc37a3c-9ad5-48c3-858e-31e943c27f92","Type":"ContainerStarted","Data":"f076c98600c79f985c45410bc910bee079f85bb5e04333993830fdf516723ee6"}
Sep 16 19:40:04.297718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.297657    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kxm92" podStartSLOduration=10.076439635 podStartE2EDuration="35.297634823s" totalImagesPullingTime="25.221195188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:04.296199945 +0000 UTC m=+35.853545042" watchObservedRunningTime="2026-09-16 19:40:04.297634823 +0000 UTC m=+35.854979911"
Sep 16 19:40:04.429415 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.429296    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:04.485839 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.485802    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:04.599103 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:04.599064    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:04.599273 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:04.599251    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:04.599347 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:04.599335    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls podName:01b5832d-8fc0-467b-b874-1a22e958fb24 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:06.599314055 +0000 UTC m=+38.156659121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls") pod "dns-default-5qm2f" (UID: "01b5832d-8fc0-467b-b874-1a22e958fb24") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:06.276615 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:06.276574    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vzhbl" event={"ID":"025c82f2-833e-4969-8602-ef8b0c6c8b64","Type":"ContainerStarted","Data":"bb1b49cd6fa6d2e8d64b98d3dcd8deee1960d3c68008c2470498538293cb28a8"}
Sep 16 19:40:06.296511 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:06.296458    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vzhbl" podStartSLOduration=1.9400404949999999 podStartE2EDuration="4.296441377s" totalImagesPullingTime="2.356400882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:06.294920758 +0000 UTC m=+37.852265860" watchObservedRunningTime="2026-09-16 19:40:06.296441377 +0000 UTC m=+37.853786463"
Sep 16 19:40:06.613318 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:06.613278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:06.613488 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:06.613447    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:06.613529 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:06.613515    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls podName:01b5832d-8fc0-467b-b874-1a22e958fb24 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:10.613498146 +0000 UTC m=+42.170843215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls") pod "dns-default-5qm2f" (UID: "01b5832d-8fc0-467b-b874-1a22e958fb24") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:07.064664 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:07.064574    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:07.166711 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:07.166677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:07.939590 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:07.939553    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:08.003535 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:08.003499    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:10.261817 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:10.261784    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:10.329298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:10.329269    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:10.400454 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:10.400424    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:10.469883 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:10.469855    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:10.647076 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:10.647045    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:10.647239 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:10.647159    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:10.647239 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:10.647211    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls podName:01b5832d-8fc0-467b-b874-1a22e958fb24 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:18.647198215 +0000 UTC m=+50.204543279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls") pod "dns-default-5qm2f" (UID: "01b5832d-8fc0-467b-b874-1a22e958fb24") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:14.041953 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:14.041909    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:14.120375 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:14.120337    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:18.700928 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:18.700893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:18.701365 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:18.701049    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:18.701365 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:18.701123    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls podName:01b5832d-8fc0-467b-b874-1a22e958fb24 nodeName:}" failed. No retries permitted until 2026-09-16 19:40:34.7011075 +0000 UTC m=+66.258452580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls") pod "dns-default-5qm2f" (UID: "01b5832d-8fc0-467b-b874-1a22e958fb24") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:30.646477 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:30.646449    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ksztl"
Sep 16 19:40:30.815998 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:30.815964    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:30.891821 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:30.891789    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:32.250942 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:32.250895    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:32.351864 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:32.351833    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:34.604024 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.603989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:40:34.606660 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.606642    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 19:40:34.614680 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:34.614663    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 19:40:34.614736 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:34.614724    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs podName:4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:38.614703205 +0000 UTC m=+130.172048275 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs") pod "network-metrics-daemon-9fgnw" (UID: "4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825") : secret "metrics-daemon-secret" not found
Sep 16 19:40:34.704778 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.704746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:40:34.704851 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.704800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:40:34.704851 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.704828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:40:34.704955 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:34.704893    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:40:34.705004 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:40:34.704965    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls podName:01b5832d-8fc0-467b-b874-1a22e958fb24 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:06.704949174 +0000 UTC m=+98.262294255 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls") pod "dns-default-5qm2f" (UID: "01b5832d-8fc0-467b-b874-1a22e958fb24") : secret "dns-default-metrics-tls" not found
Sep 16 19:40:34.707355 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.707334    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 19:40:34.707355 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.707352    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 19:40:34.717960 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.717942    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 19:40:34.719007 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.718989    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa-original-pull-secret\") pod \"global-pull-secret-syncer-jp5lb\" (UID: \"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa\") " pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:40:34.728475 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:34.728455    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf6gj\" (UniqueName: \"kubernetes.io/projected/fdccdb4c-02fe-4d84-a20a-dc9d12b896c8-kube-api-access-zf6gj\") pod \"network-check-target-vmphk\" (UID: \"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8\") " pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:40:35.005145 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:35.005067    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2djhj\""
Sep 16 19:40:35.010154 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:35.010133    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jp5lb"
Sep 16 19:40:35.012987 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:35.012971    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:40:35.286572 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:35.286493    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jp5lb"]
Sep 16 19:40:35.288913 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:35.288886    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vmphk"]
Sep 16 19:40:35.289816 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:40:35.289786    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc883cd9b_80f1_47f4_a8dc_6ae0c1eacdaa.slice/crio-0e3167d3331279d55d5398e8fb477c262e8521f5b4facc37c78fb111a3a61cba WatchSource:0}: Error finding container 0e3167d3331279d55d5398e8fb477c262e8521f5b4facc37c78fb111a3a61cba: Status 404 returned error can't find the container with id 0e3167d3331279d55d5398e8fb477c262e8521f5b4facc37c78fb111a3a61cba
Sep 16 19:40:35.292059 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:40:35.292029    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdccdb4c_02fe_4d84_a20a_dc9d12b896c8.slice/crio-d066684d56091d3e88a427778ce03e00bab6195c6948fce435a2c4d884bb545b WatchSource:0}: Error finding container d066684d56091d3e88a427778ce03e00bab6195c6948fce435a2c4d884bb545b: Status 404 returned error can't find the container with id d066684d56091d3e88a427778ce03e00bab6195c6948fce435a2c4d884bb545b
Sep 16 19:40:35.343549 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:35.343506    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vmphk" event={"ID":"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8","Type":"ContainerStarted","Data":"d066684d56091d3e88a427778ce03e00bab6195c6948fce435a2c4d884bb545b"}
Sep 16 19:40:35.344680 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:35.344658    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jp5lb" event={"ID":"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa","Type":"ContainerStarted","Data":"0e3167d3331279d55d5398e8fb477c262e8521f5b4facc37c78fb111a3a61cba"}
Sep 16 19:40:40.359731 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:40.359693    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vmphk" event={"ID":"fdccdb4c-02fe-4d84-a20a-dc9d12b896c8","Type":"ContainerStarted","Data":"35a24aeddbf8862d8af045eba920c1fc04e63cf0c61041ec3481eeb76a340e4e"}
Sep 16 19:40:40.360167 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:40.359824    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:40:40.361244 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:40.361222    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jp5lb" event={"ID":"c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa","Type":"ContainerStarted","Data":"86b6f5733af73bc6fd5b0639954ce572db259353a910fbc68dd8b5d70a7b5933"}
Sep 16 19:40:40.375315 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:40.375272    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vmphk" podStartSLOduration=67.069364946 podStartE2EDuration="1m11.375259135s" totalImagesPullingTime="4.305894189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:40.374685277 +0000 UTC m=+71.932030381" watchObservedRunningTime="2026-09-16 19:40:40.375259135 +0000 UTC m=+71.932604217"
Sep 16 19:40:40.388016 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:40.387958    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jp5lb" podStartSLOduration=67.076651066 podStartE2EDuration="1m11.387943171s" totalImagesPullingTime="4.311292105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:40.387782045 +0000 UTC m=+71.945127168" watchObservedRunningTime="2026-09-16 19:40:40.387943171 +0000 UTC m=+71.945288264"
Sep 16 19:40:49.653470 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:49.653432    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:49.730834 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:40:49.730799    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:06.726548 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:06.726495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:41:06.727039 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:06.726685    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 19:41:06.727039 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:06.726815    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls podName:01b5832d-8fc0-467b-b874-1a22e958fb24 nodeName:}" failed. No retries permitted until 2026-09-16 19:42:10.726793129 +0000 UTC m=+162.284138208 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls") pod "dns-default-5qm2f" (UID: "01b5832d-8fc0-467b-b874-1a22e958fb24") : secret "dns-default-metrics-tls" not found
Sep 16 19:41:11.366778 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:11.366747    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vmphk"
Sep 16 19:41:37.800568 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.800532    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"]
Sep 16 19:41:37.802398 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.802375    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:37.804711 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.804691    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 19:41:37.804915 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.804898    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 19:41:37.804991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.804905    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 19:41:37.805723 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.805704    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:41:37.805723 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.805719    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-974jj\""
Sep 16 19:41:37.815588 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.815567    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"]
Sep 16 19:41:37.918788 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.918756    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls"]
Sep 16 19:41:37.920424 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.920394    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b88d5d746-6tzms"]
Sep 16 19:41:37.920529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.920508    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls"
Sep 16 19:41:37.922420 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.922387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:37.922622 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.922593    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-rq9x9"]
Sep 16 19:41:37.923828 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.923812    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-g7fzt\""
Sep 16 19:41:37.924278 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.924264    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:37.924331 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.924319    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 19:41:37.924368 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.924332    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:41:37.927305 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:37.927280    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-private-configuration\" is forbidden: User \"system:node:ip-10-0-141-26.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-141-26.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret"
Sep 16 19:41:37.927305 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:37.927287    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-tls\" is forbidden: User \"system:node:ip-10-0-141-26.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-141-26.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret"
Sep 16 19:41:37.927537 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.927432    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 19:41:37.934035 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:37.934012    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"registry-dockercfg-dtnsg\" is forbidden: User \"system:node:ip-10-0-141-26.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-141-26.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dtnsg\"" type="*v1.Secret"
Sep 16 19:41:37.935567 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.935546    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:41:37.939413 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.939383    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd56b036-b438-45fa-9e25-26ab878f80da-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ggc42\" (UID: \"cd56b036-b438-45fa-9e25-26ab878f80da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:37.939482 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.939431    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhf65\" (UniqueName: \"kubernetes.io/projected/cd56b036-b438-45fa-9e25-26ab878f80da-kube-api-access-zhf65\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ggc42\" (UID: \"cd56b036-b438-45fa-9e25-26ab878f80da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:37.939520 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.939503    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd56b036-b438-45fa-9e25-26ab878f80da-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ggc42\" (UID: \"cd56b036-b438-45fa-9e25-26ab878f80da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:37.941045 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.941028    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 19:41:37.941091 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:37.941044    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-141-26.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-141-26.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap"
Sep 16 19:41:37.941241 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.941225    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 19:41:37.941350 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.941336    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qdn2d\""
Sep 16 19:41:37.942340 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.942325    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 19:41:37.946250 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.946217    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 19:41:37.946442 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.946426    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls"]
Sep 16 19:41:37.947387 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.947361    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-rq9x9"]
Sep 16 19:41:37.972640 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:37.972611    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b88d5d746-6tzms"]
Sep 16 19:41:38.040746 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040711    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqbm\" (UniqueName: \"kubernetes.io/projected/7b26210c-13af-4735-8217-764768cd4927-kube-api-access-8gqbm\") pod \"volume-data-source-validator-86695b7cc7-5gkls\" (UID: \"7b26210c-13af-4735-8217-764768cd4927\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls"
Sep 16 19:41:38.040746 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040752    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-trusted-ca\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.040989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd56b036-b438-45fa-9e25-26ab878f80da-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ggc42\" (UID: \"cd56b036-b438-45fa-9e25-26ab878f80da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:38.040989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-installation-pull-secrets\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.040989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb3b9b20-998e-47b3-a336-46904530aa4e-trusted-ca\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.040989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040841    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-bound-sa-token\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.040989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhf65\" (UniqueName: \"kubernetes.io/projected/cd56b036-b438-45fa-9e25-26ab878f80da-kube-api-access-zhf65\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ggc42\" (UID: \"cd56b036-b438-45fa-9e25-26ab878f80da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:38.040989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.040989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eb3b9b20-998e-47b3-a336-46904530aa4e-config\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.040989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3b9b20-998e-47b3-a336-46904530aa4e-serving-cert\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.040989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.040978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-certificates\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.041445 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.041006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wb6\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-kube-api-access-99wb6\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.041445 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.041088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd56b036-b438-45fa-9e25-26ab878f80da-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ggc42\" (UID: \"cd56b036-b438-45fa-9e25-26ab878f80da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:38.041445 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.041127    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-image-registry-private-configuration\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.041445 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.041158    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27e396b5-3094-448b-a33a-fbc5a9f547ca-ca-trust-extracted\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.041445 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.041209    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wpc\" (UniqueName: \"kubernetes.io/projected/eb3b9b20-998e-47b3-a336-46904530aa4e-kube-api-access-n6wpc\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.041445 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.041433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd56b036-b438-45fa-9e25-26ab878f80da-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ggc42\" (UID: \"cd56b036-b438-45fa-9e25-26ab878f80da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:38.043378 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.043358    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd56b036-b438-45fa-9e25-26ab878f80da-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ggc42\" (UID: \"cd56b036-b438-45fa-9e25-26ab878f80da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:38.056201 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.056141    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhf65\" (UniqueName: \"kubernetes.io/projected/cd56b036-b438-45fa-9e25-26ab878f80da-kube-api-access-zhf65\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ggc42\" (UID: \"cd56b036-b438-45fa-9e25-26ab878f80da\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:38.105916 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.105887    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"]
Sep 16 19:41:38.107951 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.107932    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:38.110389 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.110365    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"
Sep 16 19:41:38.111849 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.111835    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wx2pj\""
Sep 16 19:41:38.111945 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.111925    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 19:41:38.112530 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.112415    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 19:41:38.112934 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.112916    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 19:41:38.127769 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.127745    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"]
Sep 16 19:41:38.142134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-certificates\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142255 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wb6\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-kube-api-access-99wb6\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142255 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-image-registry-private-configuration\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142361 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27e396b5-3094-448b-a33a-fbc5a9f547ca-ca-trust-extracted\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142361 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wpc\" (UniqueName: \"kubernetes.io/projected/eb3b9b20-998e-47b3-a336-46904530aa4e-kube-api-access-n6wpc\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.142361 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqbm\" (UniqueName: \"kubernetes.io/projected/7b26210c-13af-4735-8217-764768cd4927-kube-api-access-8gqbm\") pod \"volume-data-source-validator-86695b7cc7-5gkls\" (UID: \"7b26210c-13af-4735-8217-764768cd4927\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls"
Sep 16 19:41:38.142531 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142439    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-trusted-ca\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142586 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-installation-pull-secrets\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142700 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142674    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27e396b5-3094-448b-a33a-fbc5a9f547ca-ca-trust-extracted\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142766 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142743    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb3b9b20-998e-47b3-a336-46904530aa4e-trusted-ca\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.142819 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142789    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-certificates\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142873 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-bound-sa-token\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142873 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.142969 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eb3b9b20-998e-47b3-a336-46904530aa4e-config\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.142969 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.142908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3b9b20-998e-47b3-a336-46904530aa4e-serving-cert\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.143711 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.143689    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eb3b9b20-998e-47b3-a336-46904530aa4e-config\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.143811 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.143709    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb3b9b20-998e-47b3-a336-46904530aa4e-trusted-ca\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.145797 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.145322    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-installation-pull-secrets\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.145944 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.145901    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb3b9b20-998e-47b3-a336-46904530aa4e-serving-cert\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.160431 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.160345    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wb6\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-kube-api-access-99wb6\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.160629 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.160580    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wpc\" (UniqueName: \"kubernetes.io/projected/eb3b9b20-998e-47b3-a336-46904530aa4e-kube-api-access-n6wpc\") pod \"console-operator-696884fb9c-rq9x9\" (UID: \"eb3b9b20-998e-47b3-a336-46904530aa4e\") " pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.160944 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.160916    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqbm\" (UniqueName: \"kubernetes.io/projected/7b26210c-13af-4735-8217-764768cd4927-kube-api-access-8gqbm\") pod \"volume-data-source-validator-86695b7cc7-5gkls\" (UID: \"7b26210c-13af-4735-8217-764768cd4927\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls"
Sep 16 19:41:38.162113 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.162093    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-bound-sa-token\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:38.186198 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.186172    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:38.230531 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.230497    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls"
Sep 16 19:41:38.241501 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.241461    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:38.243330 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.243288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:38.243330 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.243322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qv7\" (UniqueName: \"kubernetes.io/projected/e205dead-0d10-413a-97a5-24194b0bc42a-kube-api-access-z5qv7\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:38.343840 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.343813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qv7\" (UniqueName: \"kubernetes.io/projected/e205dead-0d10-413a-97a5-24194b0bc42a-kube-api-access-z5qv7\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:38.344025 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.343894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:38.344025 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:38.343991    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:41:38.344138 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:38.344055    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls podName:e205dead-0d10-413a-97a5-24194b0bc42a nodeName:}" failed. No retries permitted until 2026-09-16 19:41:38.844037567 +0000 UTC m=+130.401382653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lrttc" (UID: "e205dead-0d10-413a-97a5-24194b0bc42a") : secret "samples-operator-tls" not found
Sep 16 19:41:38.356477 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.356449    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qv7\" (UniqueName: \"kubernetes.io/projected/e205dead-0d10-413a-97a5-24194b0bc42a-kube-api-access-z5qv7\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:38.381619 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.381589    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42"]
Sep 16 19:41:38.384197 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:41:38.384174    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd56b036_b438_45fa_9e25_26ab878f80da.slice/crio-a86855af222983f61a28a30dfd303f36e1f7ade87d66999fb513b4e6b1223ff6 WatchSource:0}: Error finding container a86855af222983f61a28a30dfd303f36e1f7ade87d66999fb513b4e6b1223ff6: Status 404 returned error can't find the container with id a86855af222983f61a28a30dfd303f36e1f7ade87d66999fb513b4e6b1223ff6
Sep 16 19:41:38.503501 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.503461    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42" event={"ID":"cd56b036-b438-45fa-9e25-26ab878f80da","Type":"ContainerStarted","Data":"a86855af222983f61a28a30dfd303f36e1f7ade87d66999fb513b4e6b1223ff6"}
Sep 16 19:41:38.504632 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.504604    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls"]
Sep 16 19:41:38.507568 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:41:38.507541    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b26210c_13af_4735_8217_764768cd4927.slice/crio-aeb0befef74d86ae77338a170a6d50e4423621112686aa50808e90c51fd51ec8 WatchSource:0}: Error finding container aeb0befef74d86ae77338a170a6d50e4423621112686aa50808e90c51fd51ec8: Status 404 returned error can't find the container with id aeb0befef74d86ae77338a170a6d50e4423621112686aa50808e90c51fd51ec8
Sep 16 19:41:38.522513 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.522490    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-rq9x9"]
Sep 16 19:41:38.524935 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:41:38.524913    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3b9b20_998e_47b3_a336_46904530aa4e.slice/crio-8ffc0a100786a7043d9f8d1d820c2b92781ec279e6183a34f856317c1fd55b75 WatchSource:0}: Error finding container 8ffc0a100786a7043d9f8d1d820c2b92781ec279e6183a34f856317c1fd55b75: Status 404 returned error can't find the container with id 8ffc0a100786a7043d9f8d1d820c2b92781ec279e6183a34f856317c1fd55b75
Sep 16 19:41:38.647057 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.646964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:41:38.647202 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:38.647118    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 19:41:38.647202 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:38.647181    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs podName:4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825 nodeName:}" failed. No retries permitted until 2026-09-16 19:43:40.647165862 +0000 UTC m=+252.204510950 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs") pod "network-metrics-daemon-9fgnw" (UID: "4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825") : secret "metrics-daemon-secret" not found
Sep 16 19:41:38.848037 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.848001    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:38.848392 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:38.848150    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:41:38.848392 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:38.848214    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls podName:e205dead-0d10-413a-97a5-24194b0bc42a nodeName:}" failed. No retries permitted until 2026-09-16 19:41:39.848200357 +0000 UTC m=+131.405545438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lrttc" (UID: "e205dead-0d10-413a-97a5-24194b0bc42a") : secret "samples-operator-tls" not found
Sep 16 19:41:38.988847 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.988768    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 19:41:38.996138 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:38.996081    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-image-registry-private-configuration\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:39.005464 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.005330    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 19:41:39.013860 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.013832    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:41:39.013860 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.013862    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b88d5d746-6tzms: secret "image-registry-tls" not found
Sep 16 19:41:39.014046 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.013930    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls podName:27e396b5-3094-448b-a33a-fbc5a9f547ca nodeName:}" failed. No retries permitted until 2026-09-16 19:41:39.513909042 +0000 UTC m=+131.071254117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls") pod "image-registry-b88d5d746-6tzms" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca") : secret "image-registry-tls" not found
Sep 16 19:41:39.064309 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.063977    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dtnsg\""
Sep 16 19:41:39.143003 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.142914    2574 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:41:39.143191 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.143016    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-trusted-ca podName:27e396b5-3094-448b-a33a-fbc5a9f547ca nodeName:}" failed. No retries permitted until 2026-09-16 19:41:39.642994797 +0000 UTC m=+131.200339879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-trusted-ca") pod "image-registry-b88d5d746-6tzms" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 19:41:39.258378 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.258304    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 19:41:39.515380 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.512967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" event={"ID":"eb3b9b20-998e-47b3-a336-46904530aa4e","Type":"ContainerStarted","Data":"8ffc0a100786a7043d9f8d1d820c2b92781ec279e6183a34f856317c1fd55b75"}
Sep 16 19:41:39.516201 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.516170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls" event={"ID":"7b26210c-13af-4735-8217-764768cd4927","Type":"ContainerStarted","Data":"aeb0befef74d86ae77338a170a6d50e4423621112686aa50808e90c51fd51ec8"}
Sep 16 19:41:39.554743 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.554705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:39.554908 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.554854    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:41:39.554908 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.554869    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b88d5d746-6tzms: secret "image-registry-tls" not found
Sep 16 19:41:39.555048 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.554936    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls podName:27e396b5-3094-448b-a33a-fbc5a9f547ca nodeName:}" failed. No retries permitted until 2026-09-16 19:41:40.55491683 +0000 UTC m=+132.112261911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls") pod "image-registry-b88d5d746-6tzms" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca") : secret "image-registry-tls" not found
Sep 16 19:41:39.655186 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.655155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-trusted-ca\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:39.655973 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.655952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-trusted-ca\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:39.713132 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.713092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:39.856960 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:39.856917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:39.857435 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.857063    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:41:39.857435 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:39.857126    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls podName:e205dead-0d10-413a-97a5-24194b0bc42a nodeName:}" failed. No retries permitted until 2026-09-16 19:41:41.857108741 +0000 UTC m=+133.414453806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lrttc" (UID: "e205dead-0d10-413a-97a5-24194b0bc42a") : secret "samples-operator-tls" not found
Sep 16 19:41:40.520711 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:40.520668    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls" event={"ID":"7b26210c-13af-4735-8217-764768cd4927","Type":"ContainerStarted","Data":"9eaf4e4cb72ae1b01c7a5de72ca1eb5934af0b800f6ac3a658988708c8b40d6b"}
Sep 16 19:41:40.537186 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:40.537133    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5gkls" podStartSLOduration=2.479446292 podStartE2EDuration="3.537117553s" totalImagesPullingTime="1.057671261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:40.536792207 +0000 UTC m=+132.094137330" watchObservedRunningTime="2026-09-16 19:41:40.537117553 +0000 UTC m=+132.094462640"
Sep 16 19:41:40.562186 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:40.562156    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:40.562331 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:40.562308    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:41:40.562331 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:40.562328    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b88d5d746-6tzms: secret "image-registry-tls" not found
Sep 16 19:41:40.562423 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:40.562396    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls podName:27e396b5-3094-448b-a33a-fbc5a9f547ca nodeName:}" failed. No retries permitted until 2026-09-16 19:41:42.562374844 +0000 UTC m=+134.119719910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls") pod "image-registry-b88d5d746-6tzms" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca") : secret "image-registry-tls" not found
Sep 16 19:41:40.635575 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:40.635538    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:41.525151 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:41.525114    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42" event={"ID":"cd56b036-b438-45fa-9e25-26ab878f80da","Type":"ContainerStarted","Data":"bcd7721ae194f0852bf46fdf7ecd7c7096b814b2fdbcd5bc5f37edc1dda7a13d"}
Sep 16 19:41:41.526831 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:41.526809    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/0.log"
Sep 16 19:41:41.526940 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:41.526851    2574 generic.go:356] "Generic (PLEG): container finished" podID="eb3b9b20-998e-47b3-a336-46904530aa4e" containerID="73101e659ae2c8025eea4316084424c59e0a8ea9de4e4c8524b7267b1b7f377c" exitCode=255
Sep 16 19:41:41.527009 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:41.526935    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" event={"ID":"eb3b9b20-998e-47b3-a336-46904530aa4e","Type":"ContainerDied","Data":"73101e659ae2c8025eea4316084424c59e0a8ea9de4e4c8524b7267b1b7f377c"}
Sep 16 19:41:41.527088 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:41.527071    2574 scope.go:117] "RemoveContainer" containerID="73101e659ae2c8025eea4316084424c59e0a8ea9de4e4c8524b7267b1b7f377c"
Sep 16 19:41:41.541655 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:41.541602    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42" podStartSLOduration=2.038140844 podStartE2EDuration="4.54158831s" totalImagesPullingTime="2.503447466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:41.540911088 +0000 UTC m=+133.098256195" watchObservedRunningTime="2026-09-16 19:41:41.54158831 +0000 UTC m=+133.098933396"
Sep 16 19:41:41.872997 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:41.872960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:41.873201 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:41.873128    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:41:41.873270 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:41.873208    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls podName:e205dead-0d10-413a-97a5-24194b0bc42a nodeName:}" failed. No retries permitted until 2026-09-16 19:41:45.873187597 +0000 UTC m=+137.430532676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lrttc" (UID: "e205dead-0d10-413a-97a5-24194b0bc42a") : secret "samples-operator-tls" not found
Sep 16 19:41:42.288376 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.288263    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:42.387201 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.387168    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:42.460705 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.460672    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:42.530886 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.530861    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/1.log"
Sep 16 19:41:42.531328 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.531313    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/0.log"
Sep 16 19:41:42.531377 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.531349    2574 generic.go:356] "Generic (PLEG): container finished" podID="eb3b9b20-998e-47b3-a336-46904530aa4e" containerID="112bdf07c07c3db80bcf0f20b8ba213b4eb7c54a5baa019cf3770b9875713dc4" exitCode=255
Sep 16 19:41:42.531453 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.531437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" event={"ID":"eb3b9b20-998e-47b3-a336-46904530aa4e","Type":"ContainerDied","Data":"112bdf07c07c3db80bcf0f20b8ba213b4eb7c54a5baa019cf3770b9875713dc4"}
Sep 16 19:41:42.531508 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.531495    2574 scope.go:117] "RemoveContainer" containerID="73101e659ae2c8025eea4316084424c59e0a8ea9de4e4c8524b7267b1b7f377c"
Sep 16 19:41:42.531744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.531724    2574 scope.go:117] "RemoveContainer" containerID="112bdf07c07c3db80bcf0f20b8ba213b4eb7c54a5baa019cf3770b9875713dc4"
Sep 16 19:41:42.531936 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:42.531913    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-rq9x9_openshift-console-operator(eb3b9b20-998e-47b3-a336-46904530aa4e)\"" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" podUID="eb3b9b20-998e-47b3-a336-46904530aa4e"
Sep 16 19:41:42.578559 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.578529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:42.578683 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:42.578642    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:41:42.578683 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:42.578654    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b88d5d746-6tzms: secret "image-registry-tls" not found
Sep 16 19:41:42.578772 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:42.578706    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls podName:27e396b5-3094-448b-a33a-fbc5a9f547ca nodeName:}" failed. No retries permitted until 2026-09-16 19:41:46.578689481 +0000 UTC m=+138.136034548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls") pod "image-registry-b88d5d746-6tzms" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca") : secret "image-registry-tls" not found
Sep 16 19:41:42.808505 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:42.808423    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:43.108067 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.108029    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:43.213009 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.212971    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:43.234115 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.234083    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-42phd"]
Sep 16 19:41:43.236664 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.236647    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.241736 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.241714    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 19:41:43.241736 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.241713    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 19:41:43.242057 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.242039    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bsdbz\""
Sep 16 19:41:43.242924 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.242779    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 19:41:43.242924 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.242877    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 19:41:43.258030 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.258007    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-42phd"]
Sep 16 19:41:43.327587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.327549    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:43.385547 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.385464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbe68db7-0031-4a44-b63c-4b30698c5663-data-volume\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.385547 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.385500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.385547 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.385531    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znstm\" (UniqueName: \"kubernetes.io/projected/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-api-access-znstm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.385820 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.385644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.385820 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.385707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbe68db7-0031-4a44-b63c-4b30698c5663-crio-socket\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.486926 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.486874    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbe68db7-0031-4a44-b63c-4b30698c5663-crio-socket\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.487076 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.486995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbe68db7-0031-4a44-b63c-4b30698c5663-data-volume\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.487076 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.487012    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbe68db7-0031-4a44-b63c-4b30698c5663-crio-socket\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.487076 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.487023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.487076 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.487069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znstm\" (UniqueName: \"kubernetes.io/projected/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-api-access-znstm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.487226 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.487104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.487226 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:43.487201    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.487377 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:43.487266    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:43.987244967 +0000 UTC m=+135.544590050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.487377 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:43.487301    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.487377 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:43.487373    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:43.987355069 +0000 UTC m=+135.544700150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.487529 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.487379    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbe68db7-0031-4a44-b63c-4b30698c5663-data-volume\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.505041 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.505008    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znstm\" (UniqueName: \"kubernetes.io/projected/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-api-access-znstm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.516913 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.516881    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:43.536393 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.536371    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/1.log"
Sep 16 19:41:43.536818 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.536789    2574 scope.go:117] "RemoveContainer" containerID="112bdf07c07c3db80bcf0f20b8ba213b4eb7c54a5baa019cf3770b9875713dc4"
Sep 16 19:41:43.536997 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:43.536974    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-rq9x9_openshift-console-operator(eb3b9b20-998e-47b3-a336-46904530aa4e)\"" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" podUID="eb3b9b20-998e-47b3-a336-46904530aa4e"
Sep 16 19:41:43.588355 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.588315    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:43.990764 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.990719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.990963 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:43.990816    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:43.990963 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:43.990853    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.990963 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:43.990911    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:43.990963 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:43.990927    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:44.99091186 +0000 UTC m=+136.548256941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:43.990963 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:43.990949    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:44.990939169 +0000 UTC m=+136.548284259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:44.694940 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.694905    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hzgkb"]
Sep 16 19:41:44.697201 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.697184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:44.701193 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.701171    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 19:41:44.701304 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.701209    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 19:41:44.702208 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.702188    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wrrnz\""
Sep 16 19:41:44.702265 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.702247    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 19:41:44.702313 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.702239    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 19:41:44.712375 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.712340    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hzgkb"]
Sep 16 19:41:44.741549 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.741519    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:44.797773 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.797738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f6mz\" (UniqueName: \"kubernetes.io/projected/8bc9bc1b-5a4f-4c6a-b2bc-48be89089386-kube-api-access-6f6mz\") pod \"service-ca-568869fdc5-hzgkb\" (UID: \"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386\") " pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:44.797917 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.797841    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8bc9bc1b-5a4f-4c6a-b2bc-48be89089386-signing-key\") pod \"service-ca-568869fdc5-hzgkb\" (UID: \"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386\") " pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:44.797917 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.797875    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8bc9bc1b-5a4f-4c6a-b2bc-48be89089386-signing-cabundle\") pod \"service-ca-568869fdc5-hzgkb\" (UID: \"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386\") " pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:44.811744 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.811714    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:44.898679 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.898648    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8bc9bc1b-5a4f-4c6a-b2bc-48be89089386-signing-key\") pod \"service-ca-568869fdc5-hzgkb\" (UID: \"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386\") " pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:44.898795 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.898680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8bc9bc1b-5a4f-4c6a-b2bc-48be89089386-signing-cabundle\") pod \"service-ca-568869fdc5-hzgkb\" (UID: \"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386\") " pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:44.898795 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.898758    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f6mz\" (UniqueName: \"kubernetes.io/projected/8bc9bc1b-5a4f-4c6a-b2bc-48be89089386-kube-api-access-6f6mz\") pod \"service-ca-568869fdc5-hzgkb\" (UID: \"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386\") " pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:44.899363 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.899340    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8bc9bc1b-5a4f-4c6a-b2bc-48be89089386-signing-cabundle\") pod \"service-ca-568869fdc5-hzgkb\" (UID: \"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386\") " pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:44.901028 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.901007    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8bc9bc1b-5a4f-4c6a-b2bc-48be89089386-signing-key\") pod \"service-ca-568869fdc5-hzgkb\" (UID: \"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386\") " pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:44.908499 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.908482    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f6mz\" (UniqueName: \"kubernetes.io/projected/8bc9bc1b-5a4f-4c6a-b2bc-48be89089386-kube-api-access-6f6mz\") pod \"service-ca-568869fdc5-hzgkb\" (UID: \"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386\") " pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:45.000014 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:44.999925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:45.000160 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:45.000045    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:45.000160 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:45.000056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:45.000160 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:45.000138    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:45.000160 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:45.000158    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:47.000137406 +0000 UTC m=+138.557482498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:45.000299 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:45.000176    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:47.000164654 +0000 UTC m=+138.557509736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:45.006917 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:45.006893    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-hzgkb"
Sep 16 19:41:45.282422 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:45.282324    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-hzgkb"]
Sep 16 19:41:45.286119 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:41:45.286086    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc9bc1b_5a4f_4c6a_b2bc_48be89089386.slice/crio-7f2623985ef968a3d4be560b0b8a5bce10c68da647ed561a2d6571759bcf44c6 WatchSource:0}: Error finding container 7f2623985ef968a3d4be560b0b8a5bce10c68da647ed561a2d6571759bcf44c6: Status 404 returned error can't find the container with id 7f2623985ef968a3d4be560b0b8a5bce10c68da647ed561a2d6571759bcf44c6
Sep 16 19:41:45.370541 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:45.370518    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n6mm2_d577c619-6162-4417-8e78-f3bff26da8b7/dns-node-resolver/0.log"
Sep 16 19:41:45.544300 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:45.544220    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-hzgkb" event={"ID":"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386","Type":"ContainerStarted","Data":"7f2623985ef968a3d4be560b0b8a5bce10c68da647ed561a2d6571759bcf44c6"}
Sep 16 19:41:45.907359 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:45.907318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:45.907840 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:45.907528    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 19:41:45.907840 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:45.907610    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls podName:e205dead-0d10-413a-97a5-24194b0bc42a nodeName:}" failed. No retries permitted until 2026-09-16 19:41:53.907589083 +0000 UTC m=+145.464934164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-lrttc" (UID: "e205dead-0d10-413a-97a5-24194b0bc42a") : secret "samples-operator-tls" not found
Sep 16 19:41:45.928535 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:45.928494    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:46.369803 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:46.369772    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jdb2d_8f185472-cb62-407e-8507-df45a42c37e2/node-ca/0.log"
Sep 16 19:41:46.612945 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:46.612902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:46.613116 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:46.613080    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 19:41:46.613116 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:46.613105    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b88d5d746-6tzms: secret "image-registry-tls" not found
Sep 16 19:41:46.613205 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:46.613187    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls podName:27e396b5-3094-448b-a33a-fbc5a9f547ca nodeName:}" failed. No retries permitted until 2026-09-16 19:41:54.613170716 +0000 UTC m=+146.170515781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls") pod "image-registry-b88d5d746-6tzms" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca") : secret "image-registry-tls" not found
Sep 16 19:41:47.016998 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:47.016948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:47.017444 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:47.017025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:47.017444 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:47.017115    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:47.017444 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:47.017135    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:47.017444 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:47.017184    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:51.017169247 +0000 UTC m=+142.574514312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:47.017444 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:47.017200    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:51.017193832 +0000 UTC m=+142.574538897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:47.170134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:47.170099    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vzhbl_025c82f2-833e-4969-8602-ef8b0c6c8b64/serve-healthcheck-canary/0.log"
Sep 16 19:41:47.553363 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:47.553326    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-hzgkb" event={"ID":"8bc9bc1b-5a4f-4c6a-b2bc-48be89089386","Type":"ContainerStarted","Data":"8a89402ff8c3b7f3aae904510e49b08147d659831387dac76a737064dee72dfe"}
Sep 16 19:41:47.578890 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:47.578841    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-hzgkb" podStartSLOduration=1.799014234 podStartE2EDuration="3.578827077s" totalImagesPullingTime="1.779812843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:47.577910698 +0000 UTC m=+139.135255803" watchObservedRunningTime="2026-09-16 19:41:47.578827077 +0000 UTC m=+139.136172207"
Sep 16 19:41:47.831910 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:47.831874    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:47.931967 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:47.931932    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:48.031622 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:48.031584    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:48.135567 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:48.135456    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:48.237115 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:48.237077    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:48.241856 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:48.241821    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:48.241962 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:48.241885    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:41:48.242296 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:48.242282    2574 scope.go:117] "RemoveContainer" containerID="112bdf07c07c3db80bcf0f20b8ba213b4eb7c54a5baa019cf3770b9875713dc4"
Sep 16 19:41:48.242515 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:48.242494    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-rq9x9_openshift-console-operator(eb3b9b20-998e-47b3-a336-46904530aa4e)\"" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" podUID="eb3b9b20-998e-47b3-a336-46904530aa4e"
Sep 16 19:41:48.735731 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:48.735691    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:48.795597 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:48.795569    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:49.519584 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:49.519549    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:49.959913 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:49.959873    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:50.129026 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:50.128997    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:50.334469 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:50.334434    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:50.529004 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:50.528967    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:51.049709 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:51.049662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:51.049709 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:51.049728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:51.049968 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:51.049819    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:51.049968 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:51.049835    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 19:41:51.049968 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:51.049883    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:59.049868332 +0000 UTC m=+150.607213411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : secret "insights-runtime-extractor-tls" not found
Sep 16 19:41:51.049968 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:51.049897    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm podName:cbe68db7-0031-4a44-b63c-4b30698c5663 nodeName:}" failed. No retries permitted until 2026-09-16 19:41:59.049891273 +0000 UTC m=+150.607236338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42phd" (UID: "cbe68db7-0031-4a44-b63c-4b30698c5663") : configmap "kube-rbac-proxy" not found
Sep 16 19:41:51.323578 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:51.323486    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:51.725263 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:51.725225    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:51.794462 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:51.794428    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:52.127194 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:52.127152    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:52.333611 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:52.333578    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:52.406889 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:52.406798    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:52.737441 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:52.737333    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:53.529585 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:53.529554    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:53.921027 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:53.920988    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:53.974796 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:53.974764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:53.977064 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:53.977033    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e205dead-0d10-413a-97a5-24194b0bc42a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-lrttc\" (UID: \"e205dead-0d10-413a-97a5-24194b0bc42a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:54.017119 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:54.017084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"
Sep 16 19:41:54.133768 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:54.133733    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:54.288329 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:54.288256    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc"]
Sep 16 19:41:54.328365 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:54.328327    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:54.519244 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:54.519198    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:54.577458 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:54.577374    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc" event={"ID":"e205dead-0d10-413a-97a5-24194b0bc42a","Type":"ContainerStarted","Data":"8d3a54e354868d31a76894bb842bb2ac93e797242840e2843174263453dfe1f9"}
Sep 16 19:41:54.682310 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:54.682270    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:54.684726 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:54.684703    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") pod \"image-registry-b88d5d746-6tzms\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") " pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:54.736795 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:54.736758    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:55.024320 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:55.024047    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b88d5d746-6tzms"]
Sep 16 19:41:55.026954 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:41:55.026916    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e396b5_3094_448b_a33a_fbc5a9f547ca.slice/crio-90c51d0941ff9e37721627aad8007df1332791500d86357a59a1b11c13f81921 WatchSource:0}: Error finding container 90c51d0941ff9e37721627aad8007df1332791500d86357a59a1b11c13f81921: Status 404 returned error can't find the container with id 90c51d0941ff9e37721627aad8007df1332791500d86357a59a1b11c13f81921
Sep 16 19:41:55.357392 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:55.357338    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:55.495889 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:55.495855    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:55.582563 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:55.582525    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b88d5d746-6tzms" event={"ID":"27e396b5-3094-448b-a33a-fbc5a9f547ca","Type":"ContainerStarted","Data":"437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d"}
Sep 16 19:41:55.582723 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:55.582583    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b88d5d746-6tzms" event={"ID":"27e396b5-3094-448b-a33a-fbc5a9f547ca","Type":"ContainerStarted","Data":"90c51d0941ff9e37721627aad8007df1332791500d86357a59a1b11c13f81921"}
Sep 16 19:41:55.582723 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:55.582662    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:41:55.728010 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:55.727917    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:56.130145 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:56.130099    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:56.325281 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:56.325190    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:56.587822 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:56.587783    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc" event={"ID":"e205dead-0d10-413a-97a5-24194b0bc42a","Type":"ContainerStarted","Data":"9b2283717cbe2acf48e1957085f9516965588b2d97aaf90d0ec29a1f1c1169eb"}
Sep 16 19:41:56.588033 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:56.587841    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc" event={"ID":"e205dead-0d10-413a-97a5-24194b0bc42a","Type":"ContainerStarted","Data":"1be989d893bea74072966a4e0309d614d889179b15febf965bff48e371e0f951"}
Sep 16 19:41:56.610326 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:56.610276    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b88d5d746-6tzms" podStartSLOduration=19.610258686 podStartE2EDuration="19.610258686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:55.615544534 +0000 UTC m=+147.172889666" watchObservedRunningTime="2026-09-16 19:41:56.610258686 +0000 UTC m=+148.167603773"
Sep 16 19:41:56.610501 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:56.610470    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-lrttc" podStartSLOduration=17.032425575 podStartE2EDuration="18.610462341s" totalImagesPullingTime="1.578036766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:56.609520546 +0000 UTC m=+148.166865644" watchObservedRunningTime="2026-09-16 19:41:56.610462341 +0000 UTC m=+148.167807429"
Sep 16 19:41:56.728612 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:56.728581    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:57.558814 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:57.558768    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:57.715698 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:57.715649    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:57.780328 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:57.780288    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:57.922991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:57.922956    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:58.327780 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:58.327685    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:58.568527 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:58.568472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:59.084555 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.084527    2574 scope.go:117] "RemoveContainer" containerID="112bdf07c07c3db80bcf0f20b8ba213b4eb7c54a5baa019cf3770b9875713dc4"
Sep 16 19:41:59.121839 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.121815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:59.121950 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.121866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:59.122389 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.122366    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbe68db7-0031-4a44-b63c-4b30698c5663-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:59.124040 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.124016    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbe68db7-0031-4a44-b63c-4b30698c5663-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42phd\" (UID: \"cbe68db7-0031-4a44-b63c-4b30698c5663\") " pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:59.144993 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.144969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-42phd"
Sep 16 19:41:59.418081 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.418000    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-42phd"]
Sep 16 19:41:59.422697 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:41:59.422672    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe68db7_0031_4a44_b63c_4b30698c5663.slice/crio-b706ae49be583f9186fa46e92ba00cb1b90928b334aa3b939aa2856e2bdeb88c WatchSource:0}: Error finding container b706ae49be583f9186fa46e92ba00cb1b90928b334aa3b939aa2856e2bdeb88c: Status 404 returned error can't find the container with id b706ae49be583f9186fa46e92ba00cb1b90928b334aa3b939aa2856e2bdeb88c
Sep 16 19:41:59.598657 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.598622    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42phd" event={"ID":"cbe68db7-0031-4a44-b63c-4b30698c5663","Type":"ContainerStarted","Data":"18f0b37ff1f7caca64d6496de1e6cb09e44fd2390dc64fe731134f88fcdc406e"}
Sep 16 19:41:59.599074 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.598679    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42phd" event={"ID":"cbe68db7-0031-4a44-b63c-4b30698c5663","Type":"ContainerStarted","Data":"b706ae49be583f9186fa46e92ba00cb1b90928b334aa3b939aa2856e2bdeb88c"}
Sep 16 19:41:59.600169 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.600152    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:41:59.600563 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.600549    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/1.log"
Sep 16 19:41:59.600615 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.600585    2574 generic.go:356] "Generic (PLEG): container finished" podID="eb3b9b20-998e-47b3-a336-46904530aa4e" containerID="6ea3e345e8822d56e19bd948f2646776ce9f972c670c50c3eab3324e2f23eb45" exitCode=255
Sep 16 19:41:59.600654 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.600616    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" event={"ID":"eb3b9b20-998e-47b3-a336-46904530aa4e","Type":"ContainerDied","Data":"6ea3e345e8822d56e19bd948f2646776ce9f972c670c50c3eab3324e2f23eb45"}
Sep 16 19:41:59.600683 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.600656    2574 scope.go:117] "RemoveContainer" containerID="112bdf07c07c3db80bcf0f20b8ba213b4eb7c54a5baa019cf3770b9875713dc4"
Sep 16 19:41:59.600979 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:41:59.600962    2574 scope.go:117] "RemoveContainer" containerID="6ea3e345e8822d56e19bd948f2646776ce9f972c670c50c3eab3324e2f23eb45"
Sep 16 19:41:59.601192 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:41:59.601171    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-rq9x9_openshift-console-operator(eb3b9b20-998e-47b3-a336-46904530aa4e)\"" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" podUID="eb3b9b20-998e-47b3-a336-46904530aa4e"
Sep 16 19:42:00.604848 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:00.604817    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:42:00.606639 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:00.606611    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42phd" event={"ID":"cbe68db7-0031-4a44-b63c-4b30698c5663","Type":"ContainerStarted","Data":"d359e7e6bcd439460e60e0de28dfd52848422b23fd2a746a953865b9fc2f5b4c"}
Sep 16 19:42:01.612117 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:01.612077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42phd" event={"ID":"cbe68db7-0031-4a44-b63c-4b30698c5663","Type":"ContainerStarted","Data":"92d675ffd072916e328df956d99494f8b2842c9013de5d2eb26fd57cad5b28d2"}
Sep 16 19:42:01.637610 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:01.637557    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-42phd" podStartSLOduration=16.746796556 podStartE2EDuration="18.637540301s" totalImagesPullingTime="1.890743745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:41:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:01.635962031 +0000 UTC m=+153.193307137" watchObservedRunningTime="2026-09-16 19:42:01.637540301 +0000 UTC m=+153.194885393"
Sep 16 19:42:02.928388 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:02.928353    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:03.029202 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:03.029168    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:05.866671 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:42:05.866629    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-5qm2f" podUID="01b5832d-8fc0-467b-b874-1a22e958fb24"
Sep 16 19:42:06.185116 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.185090    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b88d5d746-6tzms"]
Sep 16 19:42:06.221916 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.221883    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:06.251314 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.251277    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d5d7d58c7-6q5zw"]
Sep 16 19:42:06.253389 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.253372    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.275651 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.275627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1287efa8-a543-4625-b434-b92369c9c632-registry-tls\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.275796 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.275662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1287efa8-a543-4625-b434-b92369c9c632-ca-trust-extracted\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.275796 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.275692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1287efa8-a543-4625-b434-b92369c9c632-installation-pull-secrets\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.275796 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.275756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1287efa8-a543-4625-b434-b92369c9c632-bound-sa-token\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.275950 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.275810    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1287efa8-a543-4625-b434-b92369c9c632-image-registry-private-configuration\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.275950 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.275839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1287efa8-a543-4625-b434-b92369c9c632-trusted-ca\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.275950 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.275881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc64f\" (UniqueName: \"kubernetes.io/projected/1287efa8-a543-4625-b434-b92369c9c632-kube-api-access-vc64f\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.275950 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.275914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1287efa8-a543-4625-b434-b92369c9c632-registry-certificates\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.289496 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.289473    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d5d7d58c7-6q5zw"]
Sep 16 19:42:06.326184 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.326148    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:06.377203 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.377167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1287efa8-a543-4625-b434-b92369c9c632-image-registry-private-configuration\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.377203 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.377206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1287efa8-a543-4625-b434-b92369c9c632-trusted-ca\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.377440 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.377227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc64f\" (UniqueName: \"kubernetes.io/projected/1287efa8-a543-4625-b434-b92369c9c632-kube-api-access-vc64f\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.377440 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.377254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1287efa8-a543-4625-b434-b92369c9c632-registry-certificates\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.377440 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.377304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1287efa8-a543-4625-b434-b92369c9c632-registry-tls\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.377440 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.377340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1287efa8-a543-4625-b434-b92369c9c632-ca-trust-extracted\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.377440 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.377370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1287efa8-a543-4625-b434-b92369c9c632-installation-pull-secrets\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.377643 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.377446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1287efa8-a543-4625-b434-b92369c9c632-bound-sa-token\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.378027 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.377994    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1287efa8-a543-4625-b434-b92369c9c632-ca-trust-extracted\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.378298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.378272    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1287efa8-a543-4625-b434-b92369c9c632-registry-certificates\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.378647 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.378626    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1287efa8-a543-4625-b434-b92369c9c632-trusted-ca\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.379775 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.379738    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1287efa8-a543-4625-b434-b92369c9c632-installation-pull-secrets\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.380021 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.380001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1287efa8-a543-4625-b434-b92369c9c632-image-registry-private-configuration\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.380057 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.380010    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1287efa8-a543-4625-b434-b92369c9c632-registry-tls\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.389475 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.389452    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc64f\" (UniqueName: \"kubernetes.io/projected/1287efa8-a543-4625-b434-b92369c9c632-kube-api-access-vc64f\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.389929 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.389911    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1287efa8-a543-4625-b434-b92369c9c632-bound-sa-token\") pod \"image-registry-d5d7d58c7-6q5zw\" (UID: \"1287efa8-a543-4625-b434-b92369c9c632\") " pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.562444 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.562341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:06.628470 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.628394    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:42:06.838355 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:06.838327    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d5d7d58c7-6q5zw"]
Sep 16 19:42:06.841200 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:42:06.841161    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1287efa8_a543_4625_b434_b92369c9c632.slice/crio-b5b549244862778c6963b622e48600ac0af56c66f95afb1939c2371220524aa9 WatchSource:0}: Error finding container b5b549244862778c6963b622e48600ac0af56c66f95afb1939c2371220524aa9: Status 404 returned error can't find the container with id b5b549244862778c6963b622e48600ac0af56c66f95afb1939c2371220524aa9
Sep 16 19:42:07.116553 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:42:07.116468    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9fgnw" podUID="4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825"
Sep 16 19:42:07.632819 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:07.632784    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw" event={"ID":"1287efa8-a543-4625-b434-b92369c9c632","Type":"ContainerStarted","Data":"3a0c14bdb06c8b35be5f77d30cd332cdc204e720b8af9396459cc72acb590208"}
Sep 16 19:42:07.632997 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:07.632833    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw" event={"ID":"1287efa8-a543-4625-b434-b92369c9c632","Type":"ContainerStarted","Data":"b5b549244862778c6963b622e48600ac0af56c66f95afb1939c2371220524aa9"}
Sep 16 19:42:07.632997 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:07.632893    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:07.701675 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:07.701621    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw" podStartSLOduration=1.701605652 podStartE2EDuration="1.701605652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:07.701560644 +0000 UTC m=+159.258905753" watchObservedRunningTime="2026-09-16 19:42:07.701605652 +0000 UTC m=+159.258950740"
Sep 16 19:42:08.242398 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:08.242358    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:42:08.242783 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:08.242432    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:42:08.242783 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:08.242768    2574 scope.go:117] "RemoveContainer" containerID="6ea3e345e8822d56e19bd948f2646776ce9f972c670c50c3eab3324e2f23eb45"
Sep 16 19:42:08.242937 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:42:08.242918    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-rq9x9_openshift-console-operator(eb3b9b20-998e-47b3-a336-46904530aa4e)\"" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" podUID="eb3b9b20-998e-47b3-a336-46904530aa4e"
Sep 16 19:42:08.622958 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:08.622924    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:08.722856 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:08.722824    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:10.810822 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:10.810773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:42:10.813133 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:10.813112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01b5832d-8fc0-467b-b874-1a22e958fb24-metrics-tls\") pod \"dns-default-5qm2f\" (UID: \"01b5832d-8fc0-467b-b874-1a22e958fb24\") " pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:42:10.831475 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:10.831447    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdb5m\""
Sep 16 19:42:10.839507 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:10.839487    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:42:11.130178 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:11.130138    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5qm2f"]
Sep 16 19:42:11.134016 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:42:11.133991    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b5832d_8fc0_467b_b874_1a22e958fb24.slice/crio-9dcb4660c681cef7772737ce0c62645f7a63c2540cf59fd9f09f655cd9f3bee2 WatchSource:0}: Error finding container 9dcb4660c681cef7772737ce0c62645f7a63c2540cf59fd9f09f655cd9f3bee2: Status 404 returned error can't find the container with id 9dcb4660c681cef7772737ce0c62645f7a63c2540cf59fd9f09f655cd9f3bee2
Sep 16 19:42:11.647835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:11.647794    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qm2f" event={"ID":"01b5832d-8fc0-467b-b874-1a22e958fb24","Type":"ContainerStarted","Data":"9dcb4660c681cef7772737ce0c62645f7a63c2540cf59fd9f09f655cd9f3bee2"}
Sep 16 19:42:12.185959 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:12.185923    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:12.297756 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:12.297708    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:12.384749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:12.384701    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:12.835625 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:12.835595    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:13.656348 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:13.656316    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qm2f" event={"ID":"01b5832d-8fc0-467b-b874-1a22e958fb24","Type":"ContainerStarted","Data":"c385f77f01e21e7bbc34426e3f2ccd7da89644b617facdfaf883c6434b3e9b00"}
Sep 16 19:42:13.656741 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:13.656364    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qm2f" event={"ID":"01b5832d-8fc0-467b-b874-1a22e958fb24","Type":"ContainerStarted","Data":"7642d49758bde5672ebbdf5bde1814d44919a31f0808df8848b12251edb5ecb6"}
Sep 16 19:42:13.656741 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:13.656462    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:42:13.677930 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:13.677853    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5qm2f" podStartSLOduration=130.160082209 podStartE2EDuration="2m11.677834432s" totalImagesPullingTime="1.517752223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:13.676937291 +0000 UTC m=+165.234282394" watchObservedRunningTime="2026-09-16 19:42:13.677834432 +0000 UTC m=+165.235179518"
Sep 16 19:42:16.190917 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.190889    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:42:16.438653 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.438614    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:16.501327 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.501251    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"]
Sep 16 19:42:16.527355 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.527323    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:16.542122 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.542101    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"]
Sep 16 19:42:16.542233 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.542210    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.545245 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.545225    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 19:42:16.546175 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.546160    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 19:42:16.546250 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.546161    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 19:42:16.546413 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.546390    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 19:42:16.547136 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.547119    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j7pgw\""
Sep 16 19:42:16.547229 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.547139    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 19:42:16.600419 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.600364    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:16.656577 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.656541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf912f13-579e-4f8f-86b1-1d7795227c93-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.656741 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.656583    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf912f13-579e-4f8f-86b1-1d7795227c93-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.656741 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.656628    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf912f13-579e-4f8f-86b1-1d7795227c93-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.656741 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.656681    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbdb\" (UniqueName: \"kubernetes.io/projected/cf912f13-579e-4f8f-86b1-1d7795227c93-kube-api-access-pdbdb\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.757774 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.757686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf912f13-579e-4f8f-86b1-1d7795227c93-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.757774 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.757736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbdb\" (UniqueName: \"kubernetes.io/projected/cf912f13-579e-4f8f-86b1-1d7795227c93-kube-api-access-pdbdb\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.757985 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.757929    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf912f13-579e-4f8f-86b1-1d7795227c93-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.758048 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.757982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf912f13-579e-4f8f-86b1-1d7795227c93-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.758624 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.758600    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf912f13-579e-4f8f-86b1-1d7795227c93-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.760124 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.760093    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf912f13-579e-4f8f-86b1-1d7795227c93-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.760249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.760210    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf912f13-579e-4f8f-86b1-1d7795227c93-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.766851 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.766829    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbdb\" (UniqueName: \"kubernetes.io/projected/cf912f13-579e-4f8f-86b1-1d7795227c93-kube-api-access-pdbdb\") pod \"prometheus-operator-68bcd7f987-9d9rg\" (UID: \"cf912f13-579e-4f8f-86b1-1d7795227c93\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:16.850839 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:16.850800    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"
Sep 16 19:42:17.166103 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:17.166071    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg"]
Sep 16 19:42:17.169227 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:42:17.169202    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf912f13_579e_4f8f_86b1_1d7795227c93.slice/crio-e93f8d36816f16b301ff2b56f6867ab0dfad5dae53bda06c1a161c8bc7af547a WatchSource:0}: Error finding container e93f8d36816f16b301ff2b56f6867ab0dfad5dae53bda06c1a161c8bc7af547a: Status 404 returned error can't find the container with id e93f8d36816f16b301ff2b56f6867ab0dfad5dae53bda06c1a161c8bc7af547a
Sep 16 19:42:17.623630 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:17.623526    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:17.671960 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:17.671925    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg" event={"ID":"cf912f13-579e-4f8f-86b1-1d7795227c93","Type":"ContainerStarted","Data":"e93f8d36816f16b301ff2b56f6867ab0dfad5dae53bda06c1a161c8bc7af547a"}
Sep 16 19:42:17.723236 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:17.723203    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:18.015542 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:18.015459    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:18.082541 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:18.082504    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:42:18.486847 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:18.486810    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:18.677338 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:18.677303    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg" event={"ID":"cf912f13-579e-4f8f-86b1-1d7795227c93","Type":"ContainerStarted","Data":"8dd8f275df10df85c45e41ebbdeae5a3cc44d2332c89d6139f87538f07a16717"}
Sep 16 19:42:18.677842 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:18.677362    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg" event={"ID":"cf912f13-579e-4f8f-86b1-1d7795227c93","Type":"ContainerStarted","Data":"cf76e7d87ab0c9809599d84c6c4d11cd32a1f2c9e066d8742d5dbb45aff8c682"}
Sep 16 19:42:18.701165 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:18.701113    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9d9rg" podStartSLOduration=1.669212426 podStartE2EDuration="2.701094448s" totalImagesPullingTime="1.031882022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:18.700119096 +0000 UTC m=+170.257464195" watchObservedRunningTime="2026-09-16 19:42:18.701094448 +0000 UTC m=+170.258439534"
Sep 16 19:42:19.337305 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:19.337274    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:19.392108 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:19.392071    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:20.612767 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.612727    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:20.706498 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.706467    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:20.816019 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.815980    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:20.883389 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.883307    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fcgb9"]
Sep 16 19:42:20.885843 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.885823    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:20.907639 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.907618    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sl9cc\""
Sep 16 19:42:20.907777 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.907656    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 19:42:20.907777 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.907686    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 19:42:20.907777 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.907730    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 19:42:20.928666 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.928636    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"]
Sep 16 19:42:20.930784 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.930770    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:20.933911 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:42:20.933886    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-tls\" is forbidden: User \"system:node:ip-10-0-141-26.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-141-26.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret"
Sep 16 19:42:20.934757 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.934740    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 19:42:20.934981 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.934968    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-slzlc\""
Sep 16 19:42:20.936101 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.936073    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 19:42:20.962096 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.962069    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:20.978224 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.978195    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"]
Sep 16 19:42:20.986595 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-wtmp\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:20.986742 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986607    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:20.986742 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986636    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpqp\" (UniqueName: \"kubernetes.io/projected/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-api-access-rkpqp\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:20.986742 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7b4b6cbf-5346-4264-874a-e299be9066b0-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:20.986742 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b9e6b6-25f6-4718-84bd-93941f725f7c-sys\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:20.986967 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986753    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:20.986967 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986790    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b6cbf-5346-4264-874a-e299be9066b0-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:20.986967 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986821    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-tls\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:20.986967 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8lv\" (UniqueName: \"kubernetes.io/projected/53b9e6b6-25f6-4718-84bd-93941f725f7c-kube-api-access-4x8lv\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:20.986967 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986890    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:20.986967 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:20.986967 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986951    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53b9e6b6-25f6-4718-84bd-93941f725f7c-root\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:20.987321 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.986985    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-textfile\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:20.987321 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.987011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:20.987321 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:20.987064    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53b9e6b6-25f6-4718-84bd-93941f725f7c-metrics-client-ca\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.041007 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.040973    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:21.083057 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.082985    2574 scope.go:117] "RemoveContainer" containerID="6ea3e345e8822d56e19bd948f2646776ce9f972c670c50c3eab3324e2f23eb45"
Sep 16 19:42:21.087617 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7b4b6cbf-5346-4264-874a-e299be9066b0-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.087700 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b9e6b6-25f6-4718-84bd-93941f725f7c-sys\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.087700 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.087700 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087693    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b6cbf-5346-4264-874a-e299be9066b0-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.087846 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-tls\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.087846 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b9e6b6-25f6-4718-84bd-93941f725f7c-sys\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.087846 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087751    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8lv\" (UniqueName: \"kubernetes.io/projected/53b9e6b6-25f6-4718-84bd-93941f725f7c-kube-api-access-4x8lv\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.087846 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.088036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.088036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087884    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53b9e6b6-25f6-4718-84bd-93941f725f7c-root\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.088036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-textfile\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.088036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.087952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.088036 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:42:21.087974    2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 19:42:21.088036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.088001    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53b9e6b6-25f6-4718-84bd-93941f725f7c-metrics-client-ca\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.088036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.088021    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/53b9e6b6-25f6-4718-84bd-93941f725f7c-root\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.088036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.088030    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-wtmp\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.088438 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:42:21.088052    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-tls podName:53b9e6b6-25f6-4718-84bd-93941f725f7c nodeName:}" failed. No retries permitted until 2026-09-16 19:42:21.588030267 +0000 UTC m=+173.145375345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-tls") pod "node-exporter-fcgb9" (UID: "53b9e6b6-25f6-4718-84bd-93941f725f7c") : secret "node-exporter-tls" not found
Sep 16 19:42:21.088438 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.088123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.088438 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.088142    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-wtmp\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.088438 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.088155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpqp\" (UniqueName: \"kubernetes.io/projected/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-api-access-rkpqp\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.088438 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.088317    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7b4b6cbf-5346-4264-874a-e299be9066b0-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.088690 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.088587    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-textfile\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.088919 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.088893    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.089108 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.089086    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53b9e6b6-25f6-4718-84bd-93941f725f7c-metrics-client-ca\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.089324 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.089302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.089553 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.089531    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b4b6cbf-5346-4264-874a-e299be9066b0-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.090718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.090693    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.091233 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.091213    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.095951 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.095906    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8lv\" (UniqueName: \"kubernetes.io/projected/53b9e6b6-25f6-4718-84bd-93941f725f7c-kube-api-access-4x8lv\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.096182 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.096087    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpqp\" (UniqueName: \"kubernetes.io/projected/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-api-access-rkpqp\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:21.140226 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.140196    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:21.249785 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.249749    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:21.398892 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.398784    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:21.496482 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.496445    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:21.592630 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.592598    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-tls\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.594861 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.594831    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/53b9e6b6-25f6-4718-84bd-93941f725f7c-node-exporter-tls\") pod \"node-exporter-fcgb9\" (UID: \"53b9e6b6-25f6-4718-84bd-93941f725f7c\") " pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.688814 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.688732    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:42:21.688814 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.688793    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" event={"ID":"eb3b9b20-998e-47b3-a336-46904530aa4e","Type":"ContainerStarted","Data":"73f5d915bbc44399e4fddf4fc63de909b19f734e530661060e965ee404615b25"}
Sep 16 19:42:21.689303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.689058    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:42:21.707195 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.707149    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9" podStartSLOduration=42.34589144 podStartE2EDuration="44.707108933s" totalImagesPullingTime="2.361217493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:21.706353077 +0000 UTC m=+173.263698189" watchObservedRunningTime="2026-09-16 19:42:21.707108933 +0000 UTC m=+173.264454019"
Sep 16 19:42:21.794446 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.794045    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcgb9"
Sep 16 19:42:21.812011 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.811986    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:21.922764 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:21.922724    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:22.012814 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.012728    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:22.055840 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.055810    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-rq9x9"
Sep 16 19:42:22.088993 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:42:22.088240    2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: failed to sync secret cache: timed out waiting for the condition
Sep 16 19:42:22.088993 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:42:22.088336    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-tls podName:7b4b6cbf-5346-4264-874a-e299be9066b0 nodeName:}" failed. No retries permitted until 2026-09-16 19:42:22.588313861 +0000 UTC m=+174.145658953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-2jj6n" (UID: "7b4b6cbf-5346-4264-874a-e299be9066b0") : failed to sync secret cache: timed out waiting for the condition
Sep 16 19:42:22.188726 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.188689    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:22.236044 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.236013    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 19:42:22.286797 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.286706    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:22.351752 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.351684    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:22.578847 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.578763    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:22.600853 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.600817    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:22.603623 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.603594    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b4b6cbf-5346-4264-874a-e299be9066b0-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-2jj6n\" (UID: \"7b4b6cbf-5346-4264-874a-e299be9066b0\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:22.693750 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.693696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcgb9" event={"ID":"53b9e6b6-25f6-4718-84bd-93941f725f7c","Type":"ContainerStarted","Data":"457254820c3524a6d058549c8bdb43500ce042f07039aaa88b63572ad33b474c"}
Sep 16 19:42:22.739368 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.739338    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"
Sep 16 19:42:22.946607 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:22.946567    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:23.040694 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:23.040658    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n"]
Sep 16 19:42:23.043474 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:42:23.043439    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4b6cbf_5346_4264_874a_e299be9066b0.slice/crio-f3c32eb34b35f702aeb2cdb8c800ca1c258245b63200382c9b246e504a099c19 WatchSource:0}: Error finding container f3c32eb34b35f702aeb2cdb8c800ca1c258245b63200382c9b246e504a099c19: Status 404 returned error can't find the container with id f3c32eb34b35f702aeb2cdb8c800ca1c258245b63200382c9b246e504a099c19
Sep 16 19:42:23.043910 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:23.043886    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:23.663135 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:23.663103    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5qm2f"
Sep 16 19:42:23.701578 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:23.700860    2574 generic.go:356] "Generic (PLEG): container finished" podID="53b9e6b6-25f6-4718-84bd-93941f725f7c" containerID="dfa79168a98e45ffea997b7c02a45a9a85d48a6d91b45610ff3ec826ae212afa" exitCode=0
Sep 16 19:42:23.701578 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:23.700985    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcgb9" event={"ID":"53b9e6b6-25f6-4718-84bd-93941f725f7c","Type":"ContainerDied","Data":"dfa79168a98e45ffea997b7c02a45a9a85d48a6d91b45610ff3ec826ae212afa"}
Sep 16 19:42:23.703239 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:23.703172    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n" event={"ID":"7b4b6cbf-5346-4264-874a-e299be9066b0","Type":"ContainerStarted","Data":"f3c32eb34b35f702aeb2cdb8c800ca1c258245b63200382c9b246e504a099c19"}
Sep 16 19:42:24.021610 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.021534    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:24.118490 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.118424    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:24.220292 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.220221    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:24.321599 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.321517    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:24.421310 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.421250    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:24.530632 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.530600    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:24.621112 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.621079    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:24.709198 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.709166    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcgb9" event={"ID":"53b9e6b6-25f6-4718-84bd-93941f725f7c","Type":"ContainerStarted","Data":"46fcb89e5c36603197007badce6708dd9463bdd3bd6f99ddb6a7c9d9db66d262"}
Sep 16 19:42:24.709661 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.709223    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcgb9" event={"ID":"53b9e6b6-25f6-4718-84bd-93941f725f7c","Type":"ContainerStarted","Data":"f8d6a037745a204763335d2e93f630460fca75fbac290882946bdbb80657f97b"}
Sep 16 19:42:24.711461 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.711432    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n" event={"ID":"7b4b6cbf-5346-4264-874a-e299be9066b0","Type":"ContainerStarted","Data":"42dd1d52acb5a86cb3f91785f0e58b69d38e43840f5f7d4b199153bd6426c93c"}
Sep 16 19:42:24.711570 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.711496    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n" event={"ID":"7b4b6cbf-5346-4264-874a-e299be9066b0","Type":"ContainerStarted","Data":"aa33256baf900b75a4ab9e88412077018d58d143af533356e9c4dc34e4c5eb09"}
Sep 16 19:42:24.711570 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.711525    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n" event={"ID":"7b4b6cbf-5346-4264-874a-e299be9066b0","Type":"ContainerStarted","Data":"755e87e5ea8d91b928c7d9d3e86d4ae0cddf51e709ea8785b1e40da2fbf2d508"}
Sep 16 19:42:24.735155 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.735106    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fcgb9" podStartSLOduration=3.726753257 podStartE2EDuration="4.735088406s" totalImagesPullingTime="944.866656ms" totalInitContainerRuntime="63.468493ms" isStatefulPod=true podCreationTimestamp="2026-09-16 19:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:24.733341088 +0000 UTC m=+176.290686196" watchObservedRunningTime="2026-09-16 19:42:24.735088406 +0000 UTC m=+176.292433492"
Sep 16 19:42:24.759268 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:24.759221    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-2jj6n" podStartSLOduration=3.466138198 podStartE2EDuration="4.75920415s" totalImagesPullingTime="1.293065952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:24.757256406 +0000 UTC m=+176.314601528" watchObservedRunningTime="2026-09-16 19:42:24.75920415 +0000 UTC m=+176.316549278"
Sep 16 19:42:25.529123 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.529085    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"]
Sep 16 19:42:25.531149 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.531125    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.533742 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.533712    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 19:42:25.533879 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.533723    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 19:42:25.534619 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.534601    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4tv67\""
Sep 16 19:42:25.534809 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.534752    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 19:42:25.534900 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.534752    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 19:42:25.534966 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.534927    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c8v974p52j1on\""
Sep 16 19:42:25.541949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.541923    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"]
Sep 16 19:42:25.626587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.626547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-secret-metrics-server-client-certs\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.626587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.626590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-client-ca-bundle\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.626780 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.626616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th56f\" (UniqueName: \"kubernetes.io/projected/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-kube-api-access-th56f\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.626780 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.626671    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-metrics-server-audit-profiles\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.626780 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.626718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-secret-metrics-server-tls\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.626780 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.626760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.626902 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.626796    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-audit-log\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.639649 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.639615    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:25.727303 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.727276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-metrics-server-audit-profiles\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.727719 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.727313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-secret-metrics-server-tls\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.727719 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.727351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.727719 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.727529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-audit-log\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.727719 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.727646    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-secret-metrics-server-client-certs\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.727719 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.727675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-client-ca-bundle\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.727719 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.727701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th56f\" (UniqueName: \"kubernetes.io/projected/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-kube-api-access-th56f\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.728014 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.727899    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-audit-log\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.728170 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.728112    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.728385 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.728368    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-metrics-server-audit-profiles\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.730261 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.730238    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-secret-metrics-server-tls\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.730593 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.730574    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-secret-metrics-server-client-certs\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.730722 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.730705    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-client-ca-bundle\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.736468 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.736440    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th56f\" (UniqueName: \"kubernetes.io/projected/4a16effe-fc2f-4cf8-a2bd-aebb5a991442-kube-api-access-th56f\") pod \"metrics-server-6b8f869fd6-4vlm4\" (UID: \"4a16effe-fc2f-4cf8-a2bd-aebb5a991442\") " pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:25.756193 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.756165    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:25.842304 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:25.842275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:26.068483 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:26.068449    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:26.121389 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:26.121239    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"]
Sep 16 19:42:26.124072 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:42:26.124035    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a16effe_fc2f_4cf8_a2bd_aebb5a991442.slice/crio-fa4a1a7a8d04cdbd2248673c6328df9f366607aca4f383aa4121bc8a67c7348b WatchSource:0}: Error finding container fa4a1a7a8d04cdbd2248673c6328df9f366607aca4f383aa4121bc8a67c7348b: Status 404 returned error can't find the container with id fa4a1a7a8d04cdbd2248673c6328df9f366607aca4f383aa4121bc8a67c7348b
Sep 16 19:42:26.721076 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:26.721030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4" event={"ID":"4a16effe-fc2f-4cf8-a2bd-aebb5a991442","Type":"ContainerStarted","Data":"fa4a1a7a8d04cdbd2248673c6328df9f366607aca4f383aa4121bc8a67c7348b"}
Sep 16 19:42:27.374231 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:27.374195    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:27.727504 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:27.727381    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4" event={"ID":"4a16effe-fc2f-4cf8-a2bd-aebb5a991442","Type":"ContainerStarted","Data":"88a58c3bc9bdf1f2ea1050dcda1cb17989d49f2b2ab5bc8acfa7255ed300ab46"}
Sep 16 19:42:27.752210 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:27.752157    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4" podStartSLOduration=1.477319706 podStartE2EDuration="2.752140562s" totalImagesPullingTime="1.274820856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:27.74931055 +0000 UTC m=+179.306655659" watchObservedRunningTime="2026-09-16 19:42:27.752140562 +0000 UTC m=+179.309485655"
Sep 16 19:42:28.423604 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:28.423568    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:28.505948 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:28.505909    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:28.640299 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:28.640265    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d5d7d58c7-6q5zw"
Sep 16 19:42:28.810840 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:28.810755    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:28.878429 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:28.878386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:30.751217 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:30.751183    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:31.203493 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.203447    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b88d5d746-6tzms" podUID="27e396b5-3094-448b-a33a-fbc5a9f547ca" containerName="registry" containerID="cri-o://437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d" gracePeriod=30
Sep 16 19:42:31.250316 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.250279    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-755dc69d64-twthn"]
Sep 16 19:42:31.252247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.252229    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.254591 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.254572    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r4ls8\""
Sep 16 19:42:31.256514 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.256499    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 19:42:31.256727 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.256704    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 19:42:31.256820 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.256802    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 19:42:31.257039 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.257024    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 19:42:31.257077 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.257037    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 19:42:31.257140 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.257122    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 19:42:31.257557 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.257539    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 19:42:31.268919 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.268890    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755dc69d64-twthn"]
Sep 16 19:42:31.380073 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.380035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-serving-cert\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.380213 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.380114    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-oauth-serving-cert\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.380213 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.380152    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-oauth-config\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.380280 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.380229    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-config\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.380280 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.380263    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gx5\" (UniqueName: \"kubernetes.io/projected/183b38a1-7266-41b5-aeb7-dae56fdc903a-kube-api-access-b4gx5\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.380336 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.380288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-service-ca\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.439220 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.439193    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:42:31.481515 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.481437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-serving-cert\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.481641 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.481518    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-oauth-serving-cert\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.481641 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.481552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-oauth-config\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.481641 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.481623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-config\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.481795 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.481653    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gx5\" (UniqueName: \"kubernetes.io/projected/183b38a1-7266-41b5-aeb7-dae56fdc903a-kube-api-access-b4gx5\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.481795 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.481681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-service-ca\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.482313 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.482285    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-oauth-serving-cert\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.482313 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.482302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-config\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.482518 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.482470    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-service-ca\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.483987 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.483958    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-oauth-config\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.484066 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.484001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-serving-cert\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.505123 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.505098    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gx5\" (UniqueName: \"kubernetes.io/projected/183b38a1-7266-41b5-aeb7-dae56fdc903a-kube-api-access-b4gx5\") pod \"console-755dc69d64-twthn\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") " pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.582147 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582113    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-installation-pull-secrets\") pod \"27e396b5-3094-448b-a33a-fbc5a9f547ca\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") "
Sep 16 19:42:31.582147 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582153    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-image-registry-private-configuration\") pod \"27e396b5-3094-448b-a33a-fbc5a9f547ca\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") "
Sep 16 19:42:31.582346 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582196    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-bound-sa-token\") pod \"27e396b5-3094-448b-a33a-fbc5a9f547ca\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") "
Sep 16 19:42:31.582346 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582221    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") pod \"27e396b5-3094-448b-a33a-fbc5a9f547ca\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") "
Sep 16 19:42:31.582346 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582244    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27e396b5-3094-448b-a33a-fbc5a9f547ca-ca-trust-extracted\") pod \"27e396b5-3094-448b-a33a-fbc5a9f547ca\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") "
Sep 16 19:42:31.582346 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582271    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-trusted-ca\") pod \"27e396b5-3094-448b-a33a-fbc5a9f547ca\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") "
Sep 16 19:42:31.582346 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582346    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99wb6\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-kube-api-access-99wb6\") pod \"27e396b5-3094-448b-a33a-fbc5a9f547ca\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") "
Sep 16 19:42:31.582597 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582379    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-certificates\") pod \"27e396b5-3094-448b-a33a-fbc5a9f547ca\" (UID: \"27e396b5-3094-448b-a33a-fbc5a9f547ca\") "
Sep 16 19:42:31.583049 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582894    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "27e396b5-3094-448b-a33a-fbc5a9f547ca" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:42:31.583049 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.582986    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "27e396b5-3094-448b-a33a-fbc5a9f547ca" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:42:31.584645 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.584617    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "27e396b5-3094-448b-a33a-fbc5a9f547ca" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:31.584752 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.584692    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "27e396b5-3094-448b-a33a-fbc5a9f547ca" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:31.584825 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.584799    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "27e396b5-3094-448b-a33a-fbc5a9f547ca" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:31.584872 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.584804    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "27e396b5-3094-448b-a33a-fbc5a9f547ca" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:31.585012 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.584996    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-kube-api-access-99wb6" (OuterVolumeSpecName: "kube-api-access-99wb6") pod "27e396b5-3094-448b-a33a-fbc5a9f547ca" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca"). InnerVolumeSpecName "kube-api-access-99wb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:31.590441 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.590390    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e396b5-3094-448b-a33a-fbc5a9f547ca-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "27e396b5-3094-448b-a33a-fbc5a9f547ca" (UID: "27e396b5-3094-448b-a33a-fbc5a9f547ca"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:31.590562 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.590549    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:31.683993 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.683963    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wb6\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-kube-api-access-99wb6\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:42:31.683993 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.683990    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-certificates\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:42:31.684176 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.684002    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-installation-pull-secrets\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:42:31.684176 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.684012    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/27e396b5-3094-448b-a33a-fbc5a9f547ca-image-registry-private-configuration\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:42:31.684176 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.684022    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-bound-sa-token\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:42:31.684176 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.684031    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/27e396b5-3094-448b-a33a-fbc5a9f547ca-registry-tls\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:42:31.684176 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.684038    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/27e396b5-3094-448b-a33a-fbc5a9f547ca-ca-trust-extracted\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:42:31.684176 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.684046    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/27e396b5-3094-448b-a33a-fbc5a9f547ca-trusted-ca\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:42:31.742745 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.742659    2574 generic.go:356] "Generic (PLEG): container finished" podID="27e396b5-3094-448b-a33a-fbc5a9f547ca" containerID="437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d" exitCode=0
Sep 16 19:42:31.742745 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.742710    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b88d5d746-6tzms"
Sep 16 19:42:31.742745 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.742723    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b88d5d746-6tzms" event={"ID":"27e396b5-3094-448b-a33a-fbc5a9f547ca","Type":"ContainerDied","Data":"437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d"}
Sep 16 19:42:31.742957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.742773    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b88d5d746-6tzms" event={"ID":"27e396b5-3094-448b-a33a-fbc5a9f547ca","Type":"ContainerDied","Data":"90c51d0941ff9e37721627aad8007df1332791500d86357a59a1b11c13f81921"}
Sep 16 19:42:31.742957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.742802    2574 scope.go:117] "RemoveContainer" containerID="437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d"
Sep 16 19:42:31.751130 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.751111    2574 scope.go:117] "RemoveContainer" containerID="437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d"
Sep 16 19:42:31.751436 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:42:31.751394    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d\": container with ID starting with 437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d not found: ID does not exist" containerID="437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d"
Sep 16 19:42:31.751705 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.751447    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d"} err="failed to get container status \"437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d\": rpc error: code = NotFound desc = could not find container \"437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d\": container with ID starting with 437d0c54c1bd64e2dcc737562502c1d5db35446a83eab84521e24f8c767ba55d not found: ID does not exist"
Sep 16 19:42:31.770992 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.770967    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b88d5d746-6tzms"]
Sep 16 19:42:31.775508 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.775488    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b88d5d746-6tzms"]
Sep 16 19:42:31.867948 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:31.867899    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-755dc69d64-twthn"]
Sep 16 19:42:31.871700 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:42:31.871675    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183b38a1_7266_41b5_aeb7_dae56fdc903a.slice/crio-c349cbd25eae70fcd971f1d7419835f1f24b969d012dd011b527761c36fe72e2 WatchSource:0}: Error finding container c349cbd25eae70fcd971f1d7419835f1f24b969d012dd011b527761c36fe72e2: Status 404 returned error can't find the container with id c349cbd25eae70fcd971f1d7419835f1f24b969d012dd011b527761c36fe72e2
Sep 16 19:42:32.748191 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:32.748156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755dc69d64-twthn" event={"ID":"183b38a1-7266-41b5-aeb7-dae56fdc903a","Type":"ContainerStarted","Data":"c349cbd25eae70fcd971f1d7419835f1f24b969d012dd011b527761c36fe72e2"}
Sep 16 19:42:33.088656 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:33.088621    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27e396b5-3094-448b-a33a-fbc5a9f547ca" path="/var/lib/kubelet/pods/27e396b5-3094-448b-a33a-fbc5a9f547ca/volumes"
Sep 16 19:42:33.529499 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:33.529416    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:33.589048 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:33.588991    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:34.681873 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:34.681833    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:34.741717 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:34.741630    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:34.758247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:34.758219    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755dc69d64-twthn" event={"ID":"183b38a1-7266-41b5-aeb7-dae56fdc903a","Type":"ContainerStarted","Data":"6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8"}
Sep 16 19:42:34.777867 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:34.777818    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-755dc69d64-twthn" podStartSLOduration=1.161797021 podStartE2EDuration="3.777799343s" totalImagesPullingTime="2.616002322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:34.777188606 +0000 UTC m=+186.334533710" watchObservedRunningTime="2026-09-16 19:42:34.777799343 +0000 UTC m=+186.335144432"
Sep 16 19:42:35.756919 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:35.756879    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:35.857641 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:35.857608    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:39.564119 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.564076    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:39.626559 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.626524    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:39.754796 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.754760    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78d7666597-pwhpn"]
Sep 16 19:42:39.755057 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.755037    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27e396b5-3094-448b-a33a-fbc5a9f547ca" containerName="registry"
Sep 16 19:42:39.755108 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.755059    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27e396b5-3094-448b-a33a-fbc5a9f547ca" containerName="registry"
Sep 16 19:42:39.755108 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.755105    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="27e396b5-3094-448b-a33a-fbc5a9f547ca" containerName="registry"
Sep 16 19:42:39.757018 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.756995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.766267 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.766241    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 19:42:39.772508 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.772482    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d7666597-pwhpn"]
Sep 16 19:42:39.854371 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.854339    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-oauth-config\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.854565 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.854377    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-oauth-serving-cert\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.854565 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.854397    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pm7b\" (UniqueName: \"kubernetes.io/projected/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-kube-api-access-6pm7b\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.854565 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.854439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-service-ca\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.854565 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.854470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-serving-cert\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.854565 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.854526    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-trusted-ca-bundle\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.854730 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.854590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-config\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.955875 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.955841    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-serving-cert\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.955875 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.955882    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-trusted-ca-bundle\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.956134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.955925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-config\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.956134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.955969    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-oauth-config\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.956134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.956108    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-oauth-serving-cert\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.956284 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.956152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pm7b\" (UniqueName: \"kubernetes.io/projected/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-kube-api-access-6pm7b\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.956284 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.956185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-service-ca\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.956777 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.956751    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-oauth-serving-cert\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.957123 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.957098    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-trusted-ca-bundle\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.957232 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.957213    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-service-ca\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.957541 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.957526    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-config\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.958344 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.958323    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-oauth-config\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.958554 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.958537    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-serving-cert\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:39.966219 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:39.966193    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pm7b\" (UniqueName: \"kubernetes.io/projected/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-kube-api-access-6pm7b\") pod \"console-78d7666597-pwhpn\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") " pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:40.066120 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:40.066080    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:40.339115 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:40.339084    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d7666597-pwhpn"]
Sep 16 19:42:40.341856 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:42:40.341827    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e5eb24_0f6e_4ef6_9d07_77139c7e6675.slice/crio-79a7a660486c7afcb0324cc594073b9e91cd3c090d930f38f00d82287ac6ac43 WatchSource:0}: Error finding container 79a7a660486c7afcb0324cc594073b9e91cd3c090d930f38f00d82287ac6ac43: Status 404 returned error can't find the container with id 79a7a660486c7afcb0324cc594073b9e91cd3c090d930f38f00d82287ac6ac43
Sep 16 19:42:40.780828 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:40.780742    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d7666597-pwhpn" event={"ID":"25e5eb24-0f6e-4ef6-9d07-77139c7e6675","Type":"ContainerStarted","Data":"efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a"}
Sep 16 19:42:40.780828 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:40.780789    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d7666597-pwhpn" event={"ID":"25e5eb24-0f6e-4ef6-9d07-77139c7e6675","Type":"ContainerStarted","Data":"79a7a660486c7afcb0324cc594073b9e91cd3c090d930f38f00d82287ac6ac43"}
Sep 16 19:42:40.802293 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:40.802240    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78d7666597-pwhpn" podStartSLOduration=1.802224915 podStartE2EDuration="1.802224915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:40.800502292 +0000 UTC m=+192.357847411" watchObservedRunningTime="2026-09-16 19:42:40.802224915 +0000 UTC m=+192.359570001"
Sep 16 19:42:41.141907 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:41.141861    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:41.244448 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:41.244414    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:41.560272 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:41.560184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:41.591204 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:41.591174    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:41.591343 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:41.591229    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:41.596353 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:41.596330    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:41.623230 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:41.623192    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:41.788329 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:41.788303    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:42:42.543301 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:42.543269    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:42.640749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:42.640712    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:44.332155 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:44.332119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:44.478896 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:44.478863    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:44.576450 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:44.576389    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:44.777147 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:44.777059    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:44.878756 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:44.878721    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:45.842503 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:45.842471    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:45.842868 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:45.842535    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:42:49.685346 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:49.685307    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:49.763815 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:49.763783    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:50.067057 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:50.066956    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:50.067057 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:50.067056    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:50.072006 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:50.071986    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:50.824654 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:50.824623    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:42:50.876285 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:50.876251    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755dc69d64-twthn"]
Sep 16 19:42:51.368054 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:51.368016    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:51.430220 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:51.430182    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:51.562485 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:42:51.562448    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:04.668579 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:04.668536    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:04.732765 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:04.732730    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:05.848029 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:05.848001    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:43:05.852173 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:05.852146    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b8f869fd6-4vlm4"
Sep 16 19:43:06.886583 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:06.886492    2574 generic.go:356] "Generic (PLEG): container finished" podID="cd56b036-b438-45fa-9e25-26ab878f80da" containerID="bcd7721ae194f0852bf46fdf7ecd7c7096b814b2fdbcd5bc5f37edc1dda7a13d" exitCode=0
Sep 16 19:43:06.886583 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:06.886544    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42" event={"ID":"cd56b036-b438-45fa-9e25-26ab878f80da","Type":"ContainerDied","Data":"bcd7721ae194f0852bf46fdf7ecd7c7096b814b2fdbcd5bc5f37edc1dda7a13d"}
Sep 16 19:43:06.887088 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:06.886958    2574 scope.go:117] "RemoveContainer" containerID="bcd7721ae194f0852bf46fdf7ecd7c7096b814b2fdbcd5bc5f37edc1dda7a13d"
Sep 16 19:43:07.892141 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:07.892107    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ggc42" event={"ID":"cd56b036-b438-45fa-9e25-26ab878f80da","Type":"ContainerStarted","Data":"87f0fa89c8bd9b8199d9d8263d13437c359f183244ea4a75d0d189e44e860a0a"}
Sep 16 19:43:15.896137 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:15.896099    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-755dc69d64-twthn" podUID="183b38a1-7266-41b5-aeb7-dae56fdc903a" containerName="console" containerID="cri-o://6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8" gracePeriod=15
Sep 16 19:43:16.130991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.130966    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:43:16.256659 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.256577    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-oauth-serving-cert\") pod \"183b38a1-7266-41b5-aeb7-dae56fdc903a\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") "
Sep 16 19:43:16.256659 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.256625    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-oauth-config\") pod \"183b38a1-7266-41b5-aeb7-dae56fdc903a\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") "
Sep 16 19:43:16.256878 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.256663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-config\") pod \"183b38a1-7266-41b5-aeb7-dae56fdc903a\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") "
Sep 16 19:43:16.256878 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.256701    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4gx5\" (UniqueName: \"kubernetes.io/projected/183b38a1-7266-41b5-aeb7-dae56fdc903a-kube-api-access-b4gx5\") pod \"183b38a1-7266-41b5-aeb7-dae56fdc903a\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") "
Sep 16 19:43:16.256878 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.256719    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-service-ca\") pod \"183b38a1-7266-41b5-aeb7-dae56fdc903a\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") "
Sep 16 19:43:16.256878 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.256737    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-serving-cert\") pod \"183b38a1-7266-41b5-aeb7-dae56fdc903a\" (UID: \"183b38a1-7266-41b5-aeb7-dae56fdc903a\") "
Sep 16 19:43:16.257099 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.257075    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "183b38a1-7266-41b5-aeb7-dae56fdc903a" (UID: "183b38a1-7266-41b5-aeb7-dae56fdc903a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:16.257172 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.257107    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-config" (OuterVolumeSpecName: "console-config") pod "183b38a1-7266-41b5-aeb7-dae56fdc903a" (UID: "183b38a1-7266-41b5-aeb7-dae56fdc903a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:16.257172 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.257123    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-service-ca" (OuterVolumeSpecName: "service-ca") pod "183b38a1-7266-41b5-aeb7-dae56fdc903a" (UID: "183b38a1-7266-41b5-aeb7-dae56fdc903a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:16.258975 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.258948    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183b38a1-7266-41b5-aeb7-dae56fdc903a-kube-api-access-b4gx5" (OuterVolumeSpecName: "kube-api-access-b4gx5") pod "183b38a1-7266-41b5-aeb7-dae56fdc903a" (UID: "183b38a1-7266-41b5-aeb7-dae56fdc903a"). InnerVolumeSpecName "kube-api-access-b4gx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:16.259071 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.258953    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "183b38a1-7266-41b5-aeb7-dae56fdc903a" (UID: "183b38a1-7266-41b5-aeb7-dae56fdc903a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:16.259071 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.259026    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "183b38a1-7266-41b5-aeb7-dae56fdc903a" (UID: "183b38a1-7266-41b5-aeb7-dae56fdc903a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:16.357981 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.357940    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-oauth-serving-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:43:16.357981 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.357973    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-oauth-config\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:43:16.357981 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.357983    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-config\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:43:16.357981 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.357992    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4gx5\" (UniqueName: \"kubernetes.io/projected/183b38a1-7266-41b5-aeb7-dae56fdc903a-kube-api-access-b4gx5\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:43:16.358228 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.358000    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/183b38a1-7266-41b5-aeb7-dae56fdc903a-service-ca\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:43:16.358228 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.358010    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/183b38a1-7266-41b5-aeb7-dae56fdc903a-console-serving-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:43:16.925258 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.925217    2574 generic.go:356] "Generic (PLEG): container finished" podID="183b38a1-7266-41b5-aeb7-dae56fdc903a" containerID="6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8" exitCode=0
Sep 16 19:43:16.925258 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.925261    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-755dc69d64-twthn"
Sep 16 19:43:16.925718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.925301    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755dc69d64-twthn" event={"ID":"183b38a1-7266-41b5-aeb7-dae56fdc903a","Type":"ContainerDied","Data":"6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8"}
Sep 16 19:43:16.925718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.925351    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-755dc69d64-twthn" event={"ID":"183b38a1-7266-41b5-aeb7-dae56fdc903a","Type":"ContainerDied","Data":"c349cbd25eae70fcd971f1d7419835f1f24b969d012dd011b527761c36fe72e2"}
Sep 16 19:43:16.925718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.925372    2574 scope.go:117] "RemoveContainer" containerID="6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8"
Sep 16 19:43:16.933653 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.933636    2574 scope.go:117] "RemoveContainer" containerID="6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8"
Sep 16 19:43:16.933960 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:43:16.933942    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8\": container with ID starting with 6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8 not found: ID does not exist" containerID="6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8"
Sep 16 19:43:16.934022 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.933971    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8"} err="failed to get container status \"6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8\": rpc error: code = NotFound desc = could not find container \"6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8\": container with ID starting with 6751050527e54c77796a6f2410a4642623c008889e85836545d5b87d8dad9cc8 not found: ID does not exist"
Sep 16 19:43:16.943772 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.943750    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-755dc69d64-twthn"]
Sep 16 19:43:16.945634 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:16.945614    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-755dc69d64-twthn"]
Sep 16 19:43:17.087584 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:17.087551    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="183b38a1-7266-41b5-aeb7-dae56fdc903a" path="/var/lib/kubelet/pods/183b38a1-7266-41b5-aeb7-dae56fdc903a/volumes"
Sep 16 19:43:27.766935 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:27.766896    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:27.866425 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:27.866373    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:37.261283 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:37.261248    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:37.318048 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:37.317998    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:40.659710 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:40.659663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:43:40.661976 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:40.661952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825-metrics-certs\") pod \"network-metrics-daemon-9fgnw\" (UID: \"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825\") " pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:43:40.886339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:40.886302    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmvpx\""
Sep 16 19:43:40.894506 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:40.894482    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fgnw"
Sep 16 19:43:41.166075 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:41.166040    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9fgnw"]
Sep 16 19:43:41.169365 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:43:41.169341    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee25f30_64ca_4a7c_a3e4_d98bfbe3c825.slice/crio-2682816d6e7ae3d7302f7bd174bc2b8cfddad693c52c2a79db687f1a2d7da55b WatchSource:0}: Error finding container 2682816d6e7ae3d7302f7bd174bc2b8cfddad693c52c2a79db687f1a2d7da55b: Status 404 returned error can't find the container with id 2682816d6e7ae3d7302f7bd174bc2b8cfddad693c52c2a79db687f1a2d7da55b
Sep 16 19:43:42.017086 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:42.017046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9fgnw" event={"ID":"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825","Type":"ContainerStarted","Data":"2682816d6e7ae3d7302f7bd174bc2b8cfddad693c52c2a79db687f1a2d7da55b"}
Sep 16 19:43:43.022898 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:43.022858    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9fgnw" event={"ID":"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825","Type":"ContainerStarted","Data":"4d291f1e3962129e522e9b17dc3e8ec05c509a72f7263875e2e521df0b18545e"}
Sep 16 19:43:43.023298 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:43.022922    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9fgnw" event={"ID":"4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825","Type":"ContainerStarted","Data":"e79783cb00a75a9ec95eaf957375bf27319f3d669550f3843a4323c00afe5fd4"}
Sep 16 19:43:43.054286 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:43.054230    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9fgnw" podStartSLOduration=253.134941084 podStartE2EDuration="4m14.054212557s" totalImagesPullingTime="919.271473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:43.052776084 +0000 UTC m=+254.610121203" watchObservedRunningTime="2026-09-16 19:43:43.054212557 +0000 UTC m=+254.611557644"
Sep 16 19:43:43.243200 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:43.243164    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:43.300593 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:43.300509    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:46.216472 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:46.216437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:46.313293 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:46.313249    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:48.801063 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.801033    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-657c5c55c6-lpqcb"]
Sep 16 19:43:48.801480 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.801456    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="183b38a1-7266-41b5-aeb7-dae56fdc903a" containerName="console"
Sep 16 19:43:48.801533 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.801480    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="183b38a1-7266-41b5-aeb7-dae56fdc903a" containerName="console"
Sep 16 19:43:48.801591 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.801579    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="183b38a1-7266-41b5-aeb7-dae56fdc903a" containerName="console"
Sep 16 19:43:48.804766 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.804747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:48.814593 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.814568    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-657c5c55c6-lpqcb"]
Sep 16 19:43:48.852363 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.852327    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:48.928106 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.928064    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-oauth-config\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:48.928275 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.928119    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-serving-cert\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:48.928275 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.928207    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-service-ca\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:48.928275 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.928243    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-trusted-ca-bundle\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:48.928458 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.928307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-oauth-serving-cert\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:48.928458 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.928331    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjjjt\" (UniqueName: \"kubernetes.io/projected/a9885af7-462a-44b1-99bd-eaa6187aee67-kube-api-access-cjjjt\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:48.928458 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:48.928373    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-console-config\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.029140 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.029096    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-serving-cert\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.029140 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.029144    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-service-ca\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.029384 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.029160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-trusted-ca-bundle\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.029384 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.029200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-oauth-serving-cert\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.029384 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.029224    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjjjt\" (UniqueName: \"kubernetes.io/projected/a9885af7-462a-44b1-99bd-eaa6187aee67-kube-api-access-cjjjt\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.029384 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.029254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-console-config\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.029384 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.029319    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-oauth-config\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.030056 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.030030    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-service-ca\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.030184 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.030054    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-oauth-serving-cert\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.030262 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.030189    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-console-config\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.030322 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.030279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-trusted-ca-bundle\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.031797 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.031771    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-oauth-config\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.031904 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.031887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-serving-cert\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.038512 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.038486    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjjjt\" (UniqueName: \"kubernetes.io/projected/a9885af7-462a-44b1-99bd-eaa6187aee67-kube-api-access-cjjjt\") pod \"console-657c5c55c6-lpqcb\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") " pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.116041 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.116000    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:49.131953 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.131921    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:49.232797 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.232768    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:49.392422 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:49.392349    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-657c5c55c6-lpqcb"]
Sep 16 19:43:49.395045 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:43:49.395015    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9885af7_462a_44b1_99bd_eaa6187aee67.slice/crio-9e68705973ad15dc121ccd8eba9908896fc5ef68efcd52abfc35d59c3943dfb0 WatchSource:0}: Error finding container 9e68705973ad15dc121ccd8eba9908896fc5ef68efcd52abfc35d59c3943dfb0: Status 404 returned error can't find the container with id 9e68705973ad15dc121ccd8eba9908896fc5ef68efcd52abfc35d59c3943dfb0
Sep 16 19:43:50.050606 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:50.050566    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657c5c55c6-lpqcb" event={"ID":"a9885af7-462a-44b1-99bd-eaa6187aee67","Type":"ContainerStarted","Data":"67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b"}
Sep 16 19:43:50.050973 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:50.050659    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657c5c55c6-lpqcb" event={"ID":"a9885af7-462a-44b1-99bd-eaa6187aee67","Type":"ContainerStarted","Data":"9e68705973ad15dc121ccd8eba9908896fc5ef68efcd52abfc35d59c3943dfb0"}
Sep 16 19:43:50.069185 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:50.069124    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-657c5c55c6-lpqcb" podStartSLOduration=2.069106815 podStartE2EDuration="2.069106815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:50.068479599 +0000 UTC m=+261.625824687" watchObservedRunningTime="2026-09-16 19:43:50.069106815 +0000 UTC m=+261.626451902"
Sep 16 19:43:59.116482 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:59.116383    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:59.116482 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:59.116458    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:43:59.121136 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:43:59.121117    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:44:00.091116 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:00.091084    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:44:00.153048 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:00.153013    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d7666597-pwhpn"]
Sep 16 19:44:00.280387 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:00.280355    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:00.380423 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:00.380316    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:00.449471 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:00.449437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:04.677058 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:04.677025    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:04.779783 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:04.779745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:04.888189 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:04.888149    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:23.655045 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:23.655004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:23.720152 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:23.720119    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:25.174779 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.174741    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78d7666597-pwhpn" podUID="25e5eb24-0f6e-4ef6-9d07-77139c7e6675" containerName="console" containerID="cri-o://efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a" gracePeriod=15
Sep 16 19:44:25.412771 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.412746    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:44:25.485180 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485087    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-serving-cert\") pod \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") "
Sep 16 19:44:25.485180 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485139    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-oauth-serving-cert\") pod \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") "
Sep 16 19:44:25.485427 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485191    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-oauth-config\") pod \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") "
Sep 16 19:44:25.485427 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485225    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-service-ca\") pod \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") "
Sep 16 19:44:25.485427 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485254    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-trusted-ca-bundle\") pod \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") "
Sep 16 19:44:25.485427 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485286    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-config\") pod \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") "
Sep 16 19:44:25.485427 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485325    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pm7b\" (UniqueName: \"kubernetes.io/projected/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-kube-api-access-6pm7b\") pod \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\" (UID: \"25e5eb24-0f6e-4ef6-9d07-77139c7e6675\") "
Sep 16 19:44:25.485698 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485669    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "25e5eb24-0f6e-4ef6-9d07-77139c7e6675" (UID: "25e5eb24-0f6e-4ef6-9d07-77139c7e6675"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:44:25.485698 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485676    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-service-ca" (OuterVolumeSpecName: "service-ca") pod "25e5eb24-0f6e-4ef6-9d07-77139c7e6675" (UID: "25e5eb24-0f6e-4ef6-9d07-77139c7e6675"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:44:25.485815 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485708    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-config" (OuterVolumeSpecName: "console-config") pod "25e5eb24-0f6e-4ef6-9d07-77139c7e6675" (UID: "25e5eb24-0f6e-4ef6-9d07-77139c7e6675"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:44:25.485815 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.485714    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "25e5eb24-0f6e-4ef6-9d07-77139c7e6675" (UID: "25e5eb24-0f6e-4ef6-9d07-77139c7e6675"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:44:25.487262 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.487238    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "25e5eb24-0f6e-4ef6-9d07-77139c7e6675" (UID: "25e5eb24-0f6e-4ef6-9d07-77139c7e6675"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:44:25.487342 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.487288    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "25e5eb24-0f6e-4ef6-9d07-77139c7e6675" (UID: "25e5eb24-0f6e-4ef6-9d07-77139c7e6675"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:44:25.487459 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.487440    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-kube-api-access-6pm7b" (OuterVolumeSpecName: "kube-api-access-6pm7b") pod "25e5eb24-0f6e-4ef6-9d07-77139c7e6675" (UID: "25e5eb24-0f6e-4ef6-9d07-77139c7e6675"). InnerVolumeSpecName "kube-api-access-6pm7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:44:25.586587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.586555    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pm7b\" (UniqueName: \"kubernetes.io/projected/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-kube-api-access-6pm7b\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:44:25.586587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.586583    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-serving-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:44:25.586587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.586592    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-oauth-serving-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:44:25.586587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.586601    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-oauth-config\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:44:25.586851 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.586610    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-service-ca\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:44:25.586851 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.586618    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-trusted-ca-bundle\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:44:25.586851 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:25.586626    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25e5eb24-0f6e-4ef6-9d07-77139c7e6675-console-config\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:44:26.178782 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:26.178742    2574 generic.go:356] "Generic (PLEG): container finished" podID="25e5eb24-0f6e-4ef6-9d07-77139c7e6675" containerID="efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a" exitCode=0
Sep 16 19:44:26.178782 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:26.178784    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d7666597-pwhpn"
Sep 16 19:44:26.179254 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:26.178825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d7666597-pwhpn" event={"ID":"25e5eb24-0f6e-4ef6-9d07-77139c7e6675","Type":"ContainerDied","Data":"efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a"}
Sep 16 19:44:26.179254 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:26.178872    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d7666597-pwhpn" event={"ID":"25e5eb24-0f6e-4ef6-9d07-77139c7e6675","Type":"ContainerDied","Data":"79a7a660486c7afcb0324cc594073b9e91cd3c090d930f38f00d82287ac6ac43"}
Sep 16 19:44:26.179254 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:26.178893    2574 scope.go:117] "RemoveContainer" containerID="efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a"
Sep 16 19:44:26.187814 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:26.187796    2574 scope.go:117] "RemoveContainer" containerID="efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a"
Sep 16 19:44:26.188089 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:44:26.188071    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a\": container with ID starting with efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a not found: ID does not exist" containerID="efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a"
Sep 16 19:44:26.188143 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:26.188099    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a"} err="failed to get container status \"efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a\": rpc error: code = NotFound desc = could not find container \"efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a\": container with ID starting with efdd844190e57338932138dda34da4765452fd8b0a89030f21cc1dcdaba2b93a not found: ID does not exist"
Sep 16 19:44:26.207977 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:26.207950    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d7666597-pwhpn"]
Sep 16 19:44:26.212221 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:26.212198    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78d7666597-pwhpn"]
Sep 16 19:44:27.086714 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:27.086680    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25e5eb24-0f6e-4ef6-9d07-77139c7e6675" path="/var/lib/kubelet/pods/25e5eb24-0f6e-4ef6-9d07-77139c7e6675/volumes"
Sep 16 19:44:28.823867 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:28.823838    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:44:28.827077 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:28.827041    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:44:28.837930 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:28.837908    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 19:44:47.931502 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:47.931462    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:48.037487 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:48.037446    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:48.133824 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:48.133785    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:49.546187 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.546154    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"]
Sep 16 19:44:49.546634 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.546456    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e5eb24-0f6e-4ef6-9d07-77139c7e6675" containerName="console"
Sep 16 19:44:49.546634 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.546469    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e5eb24-0f6e-4ef6-9d07-77139c7e6675" containerName="console"
Sep 16 19:44:49.546634 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.546531    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e5eb24-0f6e-4ef6-9d07-77139c7e6675" containerName="console"
Sep 16 19:44:49.549600 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.549583    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.554633 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.554607    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 19:44:49.557924 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.557905    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zzxjc\""
Sep 16 19:44:49.558092 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.558080    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 19:44:49.571749 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.571726    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"]
Sep 16 19:44:49.670822 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.670788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gmvk\" (UniqueName: \"kubernetes.io/projected/e2fdf582-8783-433e-a601-f4d01798fa2c-kube-api-access-5gmvk\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.670993 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.670829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.670993 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.670889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.771263 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.771226    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gmvk\" (UniqueName: \"kubernetes.io/projected/e2fdf582-8783-433e-a601-f4d01798fa2c-kube-api-access-5gmvk\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.771460 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.771272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.771460 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.771298    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.771743 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.771711    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.771840 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.771727    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.782127 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.782103    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gmvk\" (UniqueName: \"kubernetes.io/projected/e2fdf582-8783-433e-a601-f4d01798fa2c-kube-api-access-5gmvk\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:49.858541 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:49.858509    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:44:50.131347 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:50.131259    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"]
Sep 16 19:44:50.135090 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:44:50.135053    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fdf582_8783_433e_a601_f4d01798fa2c.slice/crio-19fd7fcd32e3c49703d3b7a1a4893cdf326f82797929bb61d35df23ebe9b0aae WatchSource:0}: Error finding container 19fd7fcd32e3c49703d3b7a1a4893cdf326f82797929bb61d35df23ebe9b0aae: Status 404 returned error can't find the container with id 19fd7fcd32e3c49703d3b7a1a4893cdf326f82797929bb61d35df23ebe9b0aae
Sep 16 19:44:50.137082 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:50.137068    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:44:50.275649 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:50.275616    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8" event={"ID":"e2fdf582-8783-433e-a601-f4d01798fa2c","Type":"ContainerStarted","Data":"19fd7fcd32e3c49703d3b7a1a4893cdf326f82797929bb61d35df23ebe9b0aae"}
Sep 16 19:44:56.301850 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:56.301761    2574 generic.go:356] "Generic (PLEG): container finished" podID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerID="9f91c00bef1db1de35a6522cb5fc7b723fe57e16630d8918892199c8080515c7" exitCode=0
Sep 16 19:44:56.302181 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:56.301843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8" event={"ID":"e2fdf582-8783-433e-a601-f4d01798fa2c","Type":"ContainerDied","Data":"9f91c00bef1db1de35a6522cb5fc7b723fe57e16630d8918892199c8080515c7"}
Sep 16 19:44:59.316073 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:59.316033    2574 generic.go:356] "Generic (PLEG): container finished" podID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerID="47d76d1536ebd98091f9ed2f8a40f2c05e8860818af6e11ec8fcd902e1f4f1e7" exitCode=0
Sep 16 19:44:59.316459 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:44:59.316117    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8" event={"ID":"e2fdf582-8783-433e-a601-f4d01798fa2c","Type":"ContainerDied","Data":"47d76d1536ebd98091f9ed2f8a40f2c05e8860818af6e11ec8fcd902e1f4f1e7"}
Sep 16 19:45:02.755362 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:02.755325    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:06.348125 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:06.348088    2574 generic.go:356] "Generic (PLEG): container finished" podID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerID="a4e6dfaf84c4ef5097bfff49f9bd38175b8fd27177cb8920a45054d35c2cb56f" exitCode=0
Sep 16 19:45:06.348527 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:06.348156    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8" event={"ID":"e2fdf582-8783-433e-a601-f4d01798fa2c","Type":"ContainerDied","Data":"a4e6dfaf84c4ef5097bfff49f9bd38175b8fd27177cb8920a45054d35c2cb56f"}
Sep 16 19:45:07.476392 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.476359    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:45:07.636762 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.636665    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-util\") pod \"e2fdf582-8783-433e-a601-f4d01798fa2c\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") "
Sep 16 19:45:07.636762 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.636737    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-bundle\") pod \"e2fdf582-8783-433e-a601-f4d01798fa2c\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") "
Sep 16 19:45:07.636989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.636779    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gmvk\" (UniqueName: \"kubernetes.io/projected/e2fdf582-8783-433e-a601-f4d01798fa2c-kube-api-access-5gmvk\") pod \"e2fdf582-8783-433e-a601-f4d01798fa2c\" (UID: \"e2fdf582-8783-433e-a601-f4d01798fa2c\") "
Sep 16 19:45:07.637348 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.637324    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-bundle" (OuterVolumeSpecName: "bundle") pod "e2fdf582-8783-433e-a601-f4d01798fa2c" (UID: "e2fdf582-8783-433e-a601-f4d01798fa2c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:07.638924 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.638902    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fdf582-8783-433e-a601-f4d01798fa2c-kube-api-access-5gmvk" (OuterVolumeSpecName: "kube-api-access-5gmvk") pod "e2fdf582-8783-433e-a601-f4d01798fa2c" (UID: "e2fdf582-8783-433e-a601-f4d01798fa2c"). InnerVolumeSpecName "kube-api-access-5gmvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:07.640851 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.640825    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-util" (OuterVolumeSpecName: "util") pod "e2fdf582-8783-433e-a601-f4d01798fa2c" (UID: "e2fdf582-8783-433e-a601-f4d01798fa2c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:45:07.737818 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.737775    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-util\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:45:07.737818 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.737815    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf582-8783-433e-a601-f4d01798fa2c-bundle\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:45:07.737818 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:07.737825    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gmvk\" (UniqueName: \"kubernetes.io/projected/e2fdf582-8783-433e-a601-f4d01798fa2c-kube-api-access-5gmvk\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:45:08.357693 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:08.357663    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8"
Sep 16 19:45:08.357693 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:08.357672    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a4f6h8" event={"ID":"e2fdf582-8783-433e-a601-f4d01798fa2c","Type":"ContainerDied","Data":"19fd7fcd32e3c49703d3b7a1a4893cdf326f82797929bb61d35df23ebe9b0aae"}
Sep 16 19:45:08.357943 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:08.357712    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19fd7fcd32e3c49703d3b7a1a4893cdf326f82797929bb61d35df23ebe9b0aae"
Sep 16 19:45:10.762696 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:10.762648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:10.847231 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:10.847196    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:13.058551 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.058514    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"]
Sep 16 19:45:13.058943 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.058778    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerName="pull"
Sep 16 19:45:13.058943 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.058789    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerName="pull"
Sep 16 19:45:13.058943 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.058805    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerName="util"
Sep 16 19:45:13.058943 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.058810    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerName="util"
Sep 16 19:45:13.058943 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.058823    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerName="extract"
Sep 16 19:45:13.058943 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.058829    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerName="extract"
Sep 16 19:45:13.058943 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.058878    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2fdf582-8783-433e-a601-f4d01798fa2c" containerName="extract"
Sep 16 19:45:13.105838 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.105802    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"]
Sep 16 19:45:13.106034 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.106001    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:13.108990 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.108961    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 19:45:13.108990 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.108968    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-2kw8l\""
Sep 16 19:45:13.109193 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.109018    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 19:45:13.109193 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.109038    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 19:45:13.285030 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.281055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7pn\" (UniqueName: \"kubernetes.io/projected/91e91fe6-e884-42a1-a531-cf2943efaade-kube-api-access-nd7pn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zk2qx\" (UID: \"91e91fe6-e884-42a1-a531-cf2943efaade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:13.285030 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.281164    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/91e91fe6-e884-42a1-a531-cf2943efaade-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zk2qx\" (UID: \"91e91fe6-e884-42a1-a531-cf2943efaade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:13.381703 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.381667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7pn\" (UniqueName: \"kubernetes.io/projected/91e91fe6-e884-42a1-a531-cf2943efaade-kube-api-access-nd7pn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zk2qx\" (UID: \"91e91fe6-e884-42a1-a531-cf2943efaade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:13.381903 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.381726    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/91e91fe6-e884-42a1-a531-cf2943efaade-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zk2qx\" (UID: \"91e91fe6-e884-42a1-a531-cf2943efaade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:13.384064 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.384038    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/91e91fe6-e884-42a1-a531-cf2943efaade-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zk2qx\" (UID: \"91e91fe6-e884-42a1-a531-cf2943efaade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:13.394587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.394562    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7pn\" (UniqueName: \"kubernetes.io/projected/91e91fe6-e884-42a1-a531-cf2943efaade-kube-api-access-nd7pn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-zk2qx\" (UID: \"91e91fe6-e884-42a1-a531-cf2943efaade\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:13.416433 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.416388    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:13.722964 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:13.722931    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"]
Sep 16 19:45:13.724558 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:45:13.724532    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e91fe6_e884_42a1_a531_cf2943efaade.slice/crio-fc6cf92a6a5b08ce4bad0f84739baaa34893cf793ecff68bd2a249f1b7b5d765 WatchSource:0}: Error finding container fc6cf92a6a5b08ce4bad0f84739baaa34893cf793ecff68bd2a249f1b7b5d765: Status 404 returned error can't find the container with id fc6cf92a6a5b08ce4bad0f84739baaa34893cf793ecff68bd2a249f1b7b5d765
Sep 16 19:45:14.381089 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:14.381041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx" event={"ID":"91e91fe6-e884-42a1-a531-cf2943efaade","Type":"ContainerStarted","Data":"fc6cf92a6a5b08ce4bad0f84739baaa34893cf793ecff68bd2a249f1b7b5d765"}
Sep 16 19:45:17.397110 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.397073    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx" event={"ID":"91e91fe6-e884-42a1-a531-cf2943efaade","Type":"ContainerStarted","Data":"b2bd5d1fdcad338bede721687bfcd83f0cd8acfd2f6a4945212690be01126c4a"}
Sep 16 19:45:17.397549 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.397201    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:17.420862 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.420814    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx" podStartSLOduration=1.133171092 podStartE2EDuration="4.420791347s" totalImagesPullingTime="3.287620255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:17.418921354 +0000 UTC m=+348.976266444" watchObservedRunningTime="2026-09-16 19:45:17.420791347 +0000 UTC m=+348.978136435"
Sep 16 19:45:17.504119 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.504059    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:17.787487 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.787391    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:17.794715 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.794680    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-w49rd"]
Sep 16 19:45:17.814786 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.814759    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-w49rd"]
Sep 16 19:45:17.814923 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.814884    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:17.817568 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.817543    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 19:45:17.817568 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.817556    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 19:45:17.817764 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.817730    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-kk87h\""
Sep 16 19:45:17.917742 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.917710    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a7aea956-214c-4baa-aca2-bc0149c20053-certificates\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:17.917742 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.917743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a7aea956-214c-4baa-aca2-bc0149c20053-cabundle0\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:17.917951 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:17.917774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcrtk\" (UniqueName: \"kubernetes.io/projected/a7aea956-214c-4baa-aca2-bc0149c20053-kube-api-access-vcrtk\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:18.018465 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.018422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a7aea956-214c-4baa-aca2-bc0149c20053-certificates\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:18.018465 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.018458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a7aea956-214c-4baa-aca2-bc0149c20053-cabundle0\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:18.018721 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.018501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcrtk\" (UniqueName: \"kubernetes.io/projected/a7aea956-214c-4baa-aca2-bc0149c20053-kube-api-access-vcrtk\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:18.018721 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:45:18.018583    2574 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 19:45:18.018721 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:45:18.018603    2574 secret.go:281] references non-existent secret key: ca.crt
Sep 16 19:45:18.018721 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:45:18.018611    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 19:45:18.018721 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:45:18.018624    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-w49rd: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 19:45:18.018721 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:45:18.018692    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7aea956-214c-4baa-aca2-bc0149c20053-certificates podName:a7aea956-214c-4baa-aca2-bc0149c20053 nodeName:}" failed. No retries permitted until 2026-09-16 19:45:18.518676727 +0000 UTC m=+350.076021805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a7aea956-214c-4baa-aca2-bc0149c20053-certificates") pod "keda-operator-579c8958f7-w49rd" (UID: "a7aea956-214c-4baa-aca2-bc0149c20053") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 19:45:18.019301 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.019279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a7aea956-214c-4baa-aca2-bc0149c20053-cabundle0\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:18.039688 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.039615    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcrtk\" (UniqueName: \"kubernetes.io/projected/a7aea956-214c-4baa-aca2-bc0149c20053-kube-api-access-vcrtk\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:18.045329 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.045304    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"]
Sep 16 19:45:18.070587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.070549    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:18.071127 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.071109    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.075367 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.075346    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 19:45:18.075946 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.075925    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"]
Sep 16 19:45:18.119727 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.119702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.119897 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.119739    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.119897 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.119826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8fkx\" (UniqueName: \"kubernetes.io/projected/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-kube-api-access-z8fkx\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.220978 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.220942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.221147 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.221004    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fkx\" (UniqueName: \"kubernetes.io/projected/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-kube-api-access-z8fkx\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.221147 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.221051    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.221231 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:45:18.221210    2574 secret.go:281] references non-existent secret key: tls.crt
Sep 16 19:45:18.221231 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:45:18.221223    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 16 19:45:18.221318 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:45:18.221240    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj: references non-existent secret key: tls.crt
Sep 16 19:45:18.221318 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:45:18.221304    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-certificates podName:57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1 nodeName:}" failed. No retries permitted until 2026-09-16 19:45:18.72127767 +0000 UTC m=+350.278622735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-certificates") pod "keda-metrics-apiserver-77857b7b98-tfpnj" (UID: "57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1") : references non-existent secret key: tls.crt
Sep 16 19:45:18.221472 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.221448    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.230921 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.230893    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8fkx\" (UniqueName: \"kubernetes.io/projected/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-kube-api-access-z8fkx\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.234435 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.234367    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:18.353801 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.353768    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:18.507096 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.507059    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:18.523358 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.523326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a7aea956-214c-4baa-aca2-bc0149c20053-certificates\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:18.523523 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.523470    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9pkmb"]
Sep 16 19:45:18.525705 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.525684    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a7aea956-214c-4baa-aca2-bc0149c20053-certificates\") pod \"keda-operator-579c8958f7-w49rd\" (UID: \"a7aea956-214c-4baa-aca2-bc0149c20053\") " pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:18.543825 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.543801    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:18.549033 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.548999    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 19:45:18.550937 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.550913    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9pkmb"]
Sep 16 19:45:18.605451 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.605347    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:18.624454 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.624424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2592e795-ac2b-4d1e-8abe-38901ae6ee2e-certificates\") pod \"keda-admission-6c4d864b86-9pkmb\" (UID: \"2592e795-ac2b-4d1e-8abe-38901ae6ee2e\") " pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:18.624596 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.624458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jqm\" (UniqueName: \"kubernetes.io/projected/2592e795-ac2b-4d1e-8abe-38901ae6ee2e-kube-api-access-j8jqm\") pod \"keda-admission-6c4d864b86-9pkmb\" (UID: \"2592e795-ac2b-4d1e-8abe-38901ae6ee2e\") " pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:18.666094 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.666048    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:18.725253 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.725211    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:18.725890 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.725855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2592e795-ac2b-4d1e-8abe-38901ae6ee2e-certificates\") pod \"keda-admission-6c4d864b86-9pkmb\" (UID: \"2592e795-ac2b-4d1e-8abe-38901ae6ee2e\") " pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:18.726062 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.725903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jqm\" (UniqueName: \"kubernetes.io/projected/2592e795-ac2b-4d1e-8abe-38901ae6ee2e-kube-api-access-j8jqm\") pod \"keda-admission-6c4d864b86-9pkmb\" (UID: \"2592e795-ac2b-4d1e-8abe-38901ae6ee2e\") " pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:18.726062 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.725961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.728844 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.728816    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tfpnj\" (UID: \"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:18.728970 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.728945    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2592e795-ac2b-4d1e-8abe-38901ae6ee2e-certificates\") pod \"keda-admission-6c4d864b86-9pkmb\" (UID: \"2592e795-ac2b-4d1e-8abe-38901ae6ee2e\") " pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:18.738752 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.738726    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jqm\" (UniqueName: \"kubernetes.io/projected/2592e795-ac2b-4d1e-8abe-38901ae6ee2e-kube-api-access-j8jqm\") pod \"keda-admission-6c4d864b86-9pkmb\" (UID: \"2592e795-ac2b-4d1e-8abe-38901ae6ee2e\") " pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:18.785824 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.785738    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:18.857534 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.857455    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:18.984854 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:18.984817    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:19.053984 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.053921    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-w49rd"]
Sep 16 19:45:19.056948 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:45:19.056918    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7aea956_214c_4baa_aca2_bc0149c20053.slice/crio-817571ac7602b83af777ecd63d7b619ecbf45ba53b420064b50d89910c3b5c6a WatchSource:0}: Error finding container 817571ac7602b83af777ecd63d7b619ecbf45ba53b420064b50d89910c3b5c6a: Status 404 returned error can't find the container with id 817571ac7602b83af777ecd63d7b619ecbf45ba53b420064b50d89910c3b5c6a
Sep 16 19:45:19.172234 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.172126    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9pkmb"]
Sep 16 19:45:19.174862 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:45:19.174836    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2592e795_ac2b_4d1e_8abe_38901ae6ee2e.slice/crio-b5a426126c9a287ad43f6fdab11bf04d673896c4dc578a96f68a9b57a7c44947 WatchSource:0}: Error finding container b5a426126c9a287ad43f6fdab11bf04d673896c4dc578a96f68a9b57a7c44947: Status 404 returned error can't find the container with id b5a426126c9a287ad43f6fdab11bf04d673896c4dc578a96f68a9b57a7c44947
Sep 16 19:45:19.197476 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.197443    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:19.286629 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.286514    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"]
Sep 16 19:45:19.289107 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:45:19.289080    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57eebba4_2f1a_424f_aaa0_6cd5fc1f56f1.slice/crio-9d06d56c5bff7b4026ae4b39b34239417bc37e37afdbde237f4562586f1e4c49 WatchSource:0}: Error finding container 9d06d56c5bff7b4026ae4b39b34239417bc37e37afdbde237f4562586f1e4c49: Status 404 returned error can't find the container with id 9d06d56c5bff7b4026ae4b39b34239417bc37e37afdbde237f4562586f1e4c49
Sep 16 19:45:19.406795 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.406762    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-w49rd" event={"ID":"a7aea956-214c-4baa-aca2-bc0149c20053","Type":"ContainerStarted","Data":"817571ac7602b83af777ecd63d7b619ecbf45ba53b420064b50d89910c3b5c6a"}
Sep 16 19:45:19.408075 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.408039    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj" event={"ID":"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1","Type":"ContainerStarted","Data":"9d06d56c5bff7b4026ae4b39b34239417bc37e37afdbde237f4562586f1e4c49"}
Sep 16 19:45:19.409286 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.409263    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-9pkmb" event={"ID":"2592e795-ac2b-4d1e-8abe-38901ae6ee2e","Type":"ContainerStarted","Data":"b5a426126c9a287ad43f6fdab11bf04d673896c4dc578a96f68a9b57a7c44947"}
Sep 16 19:45:19.448875 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.448791    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:19.637513 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.637467    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:19.734748 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:19.734664    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:21.421757 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:21.421666    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-9pkmb" event={"ID":"2592e795-ac2b-4d1e-8abe-38901ae6ee2e","Type":"ContainerStarted","Data":"68a954fb76f7935348ca813ff2882ca1073a0584da7fd0240569bb294c2684e4"}
Sep 16 19:45:21.422204 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:21.421795    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:21.443024 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:21.442966    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-9pkmb" podStartSLOduration=1.5262228389999999 podStartE2EDuration="3.44294791s" totalImagesPullingTime="1.916725071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:21.441719562 +0000 UTC m=+352.999064686" watchObservedRunningTime="2026-09-16 19:45:21.44294791 +0000 UTC m=+353.000292997"
Sep 16 19:45:23.433394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:23.433290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-w49rd" event={"ID":"a7aea956-214c-4baa-aca2-bc0149c20053","Type":"ContainerStarted","Data":"b5993abf71d613cbd56c6fdae5e7fa5610837e6e7e7ddc5f726ef7be35f3d45e"}
Sep 16 19:45:23.433394 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:23.433361    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:45:23.434948 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:23.434923    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj" event={"ID":"57eebba4-2f1a-424f-aaa0-6cd5fc1f56f1","Type":"ContainerStarted","Data":"234e754a6d118ec20ddb01e0b68ccd7d316aff580e38a3ca0157cf9f9fd3f586"}
Sep 16 19:45:23.435086 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:23.435061    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:23.453541 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:23.453493    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-w49rd" podStartSLOduration=2.398163247 podStartE2EDuration="6.453476555s" totalImagesPullingTime="4.055313308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:23.452551539 +0000 UTC m=+355.009896664" watchObservedRunningTime="2026-09-16 19:45:23.453476555 +0000 UTC m=+355.010821641"
Sep 16 19:45:23.471025 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:23.470979    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj" podStartSLOduration=1.651715276 podStartE2EDuration="5.470960632s" totalImagesPullingTime="3.819245356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:23.469770983 +0000 UTC m=+355.027116093" watchObservedRunningTime="2026-09-16 19:45:23.470960632 +0000 UTC m=+355.028305723"
Sep 16 19:45:34.444964 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:34.444935    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tfpnj"
Sep 16 19:45:38.403027 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:38.402990    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-zk2qx"
Sep 16 19:45:42.430106 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:42.430076    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-9pkmb"
Sep 16 19:45:44.442133 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:45:44.442099    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-w49rd"
Sep 16 19:46:09.775965 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:09.775923    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:09.876729 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:09.876687    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:10.089805 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.089770    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"]
Sep 16 19:46:10.095343 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.095326    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.100067 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.100044    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 19:46:10.100191 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.100146    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zzxjc\""
Sep 16 19:46:10.100997 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.100978    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 19:46:10.115601 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.115579    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"]
Sep 16 19:46:10.164013 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.163979    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.164164 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.164024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrvlj\" (UniqueName: \"kubernetes.io/projected/10f79794-057d-49ca-84be-fdfa7faf42e5-kube-api-access-jrvlj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.164164 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.164051    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.264728 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.264696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.264893 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.264739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrvlj\" (UniqueName: \"kubernetes.io/projected/10f79794-057d-49ca-84be-fdfa7faf42e5-kube-api-access-jrvlj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.264893 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.264764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.265102 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.265085    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.265144 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.265089    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.276852 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.276828    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrvlj\" (UniqueName: \"kubernetes.io/projected/10f79794-057d-49ca-84be-fdfa7faf42e5-kube-api-access-jrvlj\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.404729 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.404641    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:10.680522 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:10.680328    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"]
Sep 16 19:46:10.682983 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:46:10.682958    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f79794_057d_49ca_84be_fdfa7faf42e5.slice/crio-59bda474ecfbc9f9b5100802c06fe0dabdf67bd089ee5be3b0916d71d11279ee WatchSource:0}: Error finding container 59bda474ecfbc9f9b5100802c06fe0dabdf67bd089ee5be3b0916d71d11279ee: Status 404 returned error can't find the container with id 59bda474ecfbc9f9b5100802c06fe0dabdf67bd089ee5be3b0916d71d11279ee
Sep 16 19:46:11.652482 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:11.652445    2574 generic.go:356] "Generic (PLEG): container finished" podID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerID="0d6dfc393b0b6a953dc79159e7a040247b12ebfa064c041cee0f01411e207ba9" exitCode=0
Sep 16 19:46:11.652962 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:11.652494    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5" event={"ID":"10f79794-057d-49ca-84be-fdfa7faf42e5","Type":"ContainerDied","Data":"0d6dfc393b0b6a953dc79159e7a040247b12ebfa064c041cee0f01411e207ba9"}
Sep 16 19:46:11.652962 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:11.652528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5" event={"ID":"10f79794-057d-49ca-84be-fdfa7faf42e5","Type":"ContainerStarted","Data":"59bda474ecfbc9f9b5100802c06fe0dabdf67bd089ee5be3b0916d71d11279ee"}
Sep 16 19:46:13.663896 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:13.663863    2574 generic.go:356] "Generic (PLEG): container finished" podID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerID="719ea43ae4366fe287a7ff28c8b7bf9ea09f6f4a9f27e6439729156febe7a876" exitCode=0
Sep 16 19:46:13.664258 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:13.663907    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5" event={"ID":"10f79794-057d-49ca-84be-fdfa7faf42e5","Type":"ContainerDied","Data":"719ea43ae4366fe287a7ff28c8b7bf9ea09f6f4a9f27e6439729156febe7a876"}
Sep 16 19:46:14.670620 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:14.670581    2574 generic.go:356] "Generic (PLEG): container finished" podID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerID="ca373481fff36e7c7f3060409d456ff6cf7e82972ad99c8d2b98ecaa29307076" exitCode=0
Sep 16 19:46:14.670986 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:14.670634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5" event={"ID":"10f79794-057d-49ca-84be-fdfa7faf42e5","Type":"ContainerDied","Data":"ca373481fff36e7c7f3060409d456ff6cf7e82972ad99c8d2b98ecaa29307076"}
Sep 16 19:46:15.797147 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:15.797115    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:15.913257 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:15.913221    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-bundle\") pod \"10f79794-057d-49ca-84be-fdfa7faf42e5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") "
Sep 16 19:46:15.913476 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:15.913274    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-util\") pod \"10f79794-057d-49ca-84be-fdfa7faf42e5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") "
Sep 16 19:46:15.913476 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:15.913336    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrvlj\" (UniqueName: \"kubernetes.io/projected/10f79794-057d-49ca-84be-fdfa7faf42e5-kube-api-access-jrvlj\") pod \"10f79794-057d-49ca-84be-fdfa7faf42e5\" (UID: \"10f79794-057d-49ca-84be-fdfa7faf42e5\") "
Sep 16 19:46:15.914191 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:15.914158    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-bundle" (OuterVolumeSpecName: "bundle") pod "10f79794-057d-49ca-84be-fdfa7faf42e5" (UID: "10f79794-057d-49ca-84be-fdfa7faf42e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:15.915501 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:15.915477    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f79794-057d-49ca-84be-fdfa7faf42e5-kube-api-access-jrvlj" (OuterVolumeSpecName: "kube-api-access-jrvlj") pod "10f79794-057d-49ca-84be-fdfa7faf42e5" (UID: "10f79794-057d-49ca-84be-fdfa7faf42e5"). InnerVolumeSpecName "kube-api-access-jrvlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:15.919008 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:15.918985    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-util" (OuterVolumeSpecName: "util") pod "10f79794-057d-49ca-84be-fdfa7faf42e5" (UID: "10f79794-057d-49ca-84be-fdfa7faf42e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:16.014484 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:16.014383    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-bundle\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:46:16.014484 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:16.014435    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10f79794-057d-49ca-84be-fdfa7faf42e5-util\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:46:16.014484 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:16.014444    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrvlj\" (UniqueName: \"kubernetes.io/projected/10f79794-057d-49ca-84be-fdfa7faf42e5-kube-api-access-jrvlj\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:46:16.681448 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:16.681418    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5"
Sep 16 19:46:16.681616 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:16.681394    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540swnh5" event={"ID":"10f79794-057d-49ca-84be-fdfa7faf42e5","Type":"ContainerDied","Data":"59bda474ecfbc9f9b5100802c06fe0dabdf67bd089ee5be3b0916d71d11279ee"}
Sep 16 19:46:16.681616 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:16.681537    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59bda474ecfbc9f9b5100802c06fe0dabdf67bd089ee5be3b0916d71d11279ee"
Sep 16 19:46:17.853459 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:17.853426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:17.952713 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:17.952683    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:21.035275 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:21.035240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:22.120189 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:22.120154    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:24.846330 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:24.846288    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:24.950003 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:24.949960    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:27.303511 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:27.303474    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:27.396618 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:27.396587    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:29.680075 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:29.680039    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:29.848444 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:29.848386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:29.939012 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:29.938921    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:30.043262 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:30.043227    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:31.287336 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:31.287298    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:31.378955 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:31.378915    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:31.478367 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:31.478329    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:32.230170 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.230134    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"]
Sep 16 19:46:32.230471 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.230457    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerName="pull"
Sep 16 19:46:32.230471 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.230472    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerName="pull"
Sep 16 19:46:32.230570 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.230481    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerName="extract"
Sep 16 19:46:32.230570 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.230486    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerName="extract"
Sep 16 19:46:32.230570 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.230496    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerName="util"
Sep 16 19:46:32.230570 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.230501    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerName="util"
Sep 16 19:46:32.230570 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.230570    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f79794-057d-49ca-84be-fdfa7faf42e5" containerName="extract"
Sep 16 19:46:32.232597 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.232578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.235411 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.235378    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 19:46:32.236351 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.236333    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 19:46:32.236468 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.236366    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zzxjc\""
Sep 16 19:46:32.242933 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.242911    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"]
Sep 16 19:46:32.345475 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.345442    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wddl\" (UniqueName: \"kubernetes.io/projected/67bf695e-5c7f-4a41-a970-d9de3ed673fd-kube-api-access-4wddl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.345821 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.345506    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.345821 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.345589    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.446177 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.446138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wddl\" (UniqueName: \"kubernetes.io/projected/67bf695e-5c7f-4a41-a970-d9de3ed673fd-kube-api-access-4wddl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.446349 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.446180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.446349 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.446243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.446626 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.446606    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.446667 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.446637    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.458381 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.458351    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wddl\" (UniqueName: \"kubernetes.io/projected/67bf695e-5c7f-4a41-a970-d9de3ed673fd-kube-api-access-4wddl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.542311 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.542229    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:32.812914 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:32.812885    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"]
Sep 16 19:46:32.815377 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:46:32.815349    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bf695e_5c7f_4a41_a970_d9de3ed673fd.slice/crio-004945a2b50b115971c7517bb785e3c9ad2c2c207e70777f5d65e36a29e88341 WatchSource:0}: Error finding container 004945a2b50b115971c7517bb785e3c9ad2c2c207e70777f5d65e36a29e88341: Status 404 returned error can't find the container with id 004945a2b50b115971c7517bb785e3c9ad2c2c207e70777f5d65e36a29e88341
Sep 16 19:46:33.760518 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:33.760437    2574 generic.go:356] "Generic (PLEG): container finished" podID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerID="33c0f47ed44de6f376821b30e1f6cc3693f876e5c464d948da8ca62a4b9cfb38" exitCode=0
Sep 16 19:46:33.760518 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:33.760499    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk" event={"ID":"67bf695e-5c7f-4a41-a970-d9de3ed673fd","Type":"ContainerDied","Data":"33c0f47ed44de6f376821b30e1f6cc3693f876e5c464d948da8ca62a4b9cfb38"}
Sep 16 19:46:33.760895 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:33.760540    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk" event={"ID":"67bf695e-5c7f-4a41-a970-d9de3ed673fd","Type":"ContainerStarted","Data":"004945a2b50b115971c7517bb785e3c9ad2c2c207e70777f5d65e36a29e88341"}
Sep 16 19:46:36.776198 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:36.776154    2574 generic.go:356] "Generic (PLEG): container finished" podID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerID="ede26783ca599f3847b21ea158fec4514e84623eb057140a021cc086c710a7ac" exitCode=0
Sep 16 19:46:36.776688 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:36.776295    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk" event={"ID":"67bf695e-5c7f-4a41-a970-d9de3ed673fd","Type":"ContainerDied","Data":"ede26783ca599f3847b21ea158fec4514e84623eb057140a021cc086c710a7ac"}
Sep 16 19:46:36.821225 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:36.821193    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:37.783530 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:37.783493    2574 generic.go:356] "Generic (PLEG): container finished" podID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerID="cd3d46eb51b702d25845a77df6ea2eba8ffe0c3e06f1de06de17ef01f0e05535" exitCode=0
Sep 16 19:46:37.783530 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:37.783532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk" event={"ID":"67bf695e-5c7f-4a41-a970-d9de3ed673fd","Type":"ContainerDied","Data":"cd3d46eb51b702d25845a77df6ea2eba8ffe0c3e06f1de06de17ef01f0e05535"}
Sep 16 19:46:38.912386 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:38.912358    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:39.005267 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.005234    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-bundle\") pod \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") "
Sep 16 19:46:39.005267 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.005272    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-util\") pod \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") "
Sep 16 19:46:39.005544 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.005292    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wddl\" (UniqueName: \"kubernetes.io/projected/67bf695e-5c7f-4a41-a970-d9de3ed673fd-kube-api-access-4wddl\") pod \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\" (UID: \"67bf695e-5c7f-4a41-a970-d9de3ed673fd\") "
Sep 16 19:46:39.005737 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.005697    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-bundle" (OuterVolumeSpecName: "bundle") pod "67bf695e-5c7f-4a41-a970-d9de3ed673fd" (UID: "67bf695e-5c7f-4a41-a970-d9de3ed673fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:39.007465 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.007441    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bf695e-5c7f-4a41-a970-d9de3ed673fd-kube-api-access-4wddl" (OuterVolumeSpecName: "kube-api-access-4wddl") pod "67bf695e-5c7f-4a41-a970-d9de3ed673fd" (UID: "67bf695e-5c7f-4a41-a970-d9de3ed673fd"). InnerVolumeSpecName "kube-api-access-4wddl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:39.010537 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.010504    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-util" (OuterVolumeSpecName: "util") pod "67bf695e-5c7f-4a41-a970-d9de3ed673fd" (UID: "67bf695e-5c7f-4a41-a970-d9de3ed673fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:39.105817 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.105787    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-bundle\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.105817 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.105812    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67bf695e-5c7f-4a41-a970-d9de3ed673fd-util\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.106003 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.105823    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wddl\" (UniqueName: \"kubernetes.io/projected/67bf695e-5c7f-4a41-a970-d9de3ed673fd-kube-api-access-4wddl\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:46:39.793940 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.793903    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk" event={"ID":"67bf695e-5c7f-4a41-a970-d9de3ed673fd","Type":"ContainerDied","Data":"004945a2b50b115971c7517bb785e3c9ad2c2c207e70777f5d65e36a29e88341"}
Sep 16 19:46:39.794111 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.793959    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="004945a2b50b115971c7517bb785e3c9ad2c2c207e70777f5d65e36a29e88341"
Sep 16 19:46:39.794111 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:39.793920    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f49mqk"
Sep 16 19:46:41.863372 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:41.863337    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:43.143624 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:43.143594    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:44.749730 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:44.749692    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:44.850038 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:44.850005    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:54.853352 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:46:54.853275    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:01.303313 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.303277    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:01.366789 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.366757    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"]
Sep 16 19:47:01.367141 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.367122    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerName="pull"
Sep 16 19:47:01.367227 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.367142    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerName="pull"
Sep 16 19:47:01.367227 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.367163    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerName="extract"
Sep 16 19:47:01.367227 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.367171    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerName="extract"
Sep 16 19:47:01.367227 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.367193    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerName="util"
Sep 16 19:47:01.367227 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.367203    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerName="util"
Sep 16 19:47:01.367490 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.367284    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="67bf695e-5c7f-4a41-a970-d9de3ed673fd" containerName="extract"
Sep 16 19:47:01.370246 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.370224    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.373026 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.373004    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:47:01.373581 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.373529    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 19:47:01.373666 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.373547    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 19:47:01.373718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.373684    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 19:47:01.373718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.373700    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 19:47:01.373991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.373974    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-crrpr\""
Sep 16 19:47:01.374070 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.373997    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 19:47:01.403266 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.403233    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"]
Sep 16 19:47:01.474550 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.474509    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:01.496935 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.496896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.497249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.496941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.497249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.496983    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.497249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.497005    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lk8j\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-kube-api-access-6lk8j\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.497249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.497070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.497249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.497118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906dfd-32df-4809-bab9-496d5dc17c2f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.497249 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.497156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.587503 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.587471    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:01.598518 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.598485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.598622 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.598531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.598670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.598647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.598714 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.598683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lk8j\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-kube-api-access-6lk8j\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.598848 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.598825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.598915 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.598895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906dfd-32df-4809-bab9-496d5dc17c2f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.599036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.598935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.599112 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.599089    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.601085 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.601054    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906dfd-32df-4809-bab9-496d5dc17c2f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.601184 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.601089    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.601184 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.601060    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.601258 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.601209    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.610617 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.610596    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lk8j\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-kube-api-access-6lk8j\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.624988 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.624961    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cwtqk\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:01.679992 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:01.679955    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:02.000186 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:47:02.000156    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4906dfd_32df_4809_bab9_496d5dc17c2f.slice/crio-914775c49f38354f5b8f22f825ad1259a73d399a178b669e9acae8497e33dec2 WatchSource:0}: Error finding container 914775c49f38354f5b8f22f825ad1259a73d399a178b669e9acae8497e33dec2: Status 404 returned error can't find the container with id 914775c49f38354f5b8f22f825ad1259a73d399a178b669e9acae8497e33dec2
Sep 16 19:47:02.008956 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:02.008929    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"]
Sep 16 19:47:02.897457 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:02.897421    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk" event={"ID":"e4906dfd-32df-4809-bab9-496d5dc17c2f","Type":"ContainerStarted","Data":"914775c49f38354f5b8f22f825ad1259a73d399a178b669e9acae8497e33dec2"}
Sep 16 19:47:03.663370 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:03.663331    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:04.209271 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:04.209224    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:47:04.209560 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:04.209314    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:47:04.909210 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:04.909173    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk" event={"ID":"e4906dfd-32df-4809-bab9-496d5dc17c2f","Type":"ContainerStarted","Data":"b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852"}
Sep 16 19:47:04.909382 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:04.909260    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:04.990790 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:04.990727    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk" podStartSLOduration=1.783950773 podStartE2EDuration="3.990705755s" totalImagesPullingTime="2.206754982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:04.984787358 +0000 UTC m=+456.542132483" watchObservedRunningTime="2026-09-16 19:47:04.990705755 +0000 UTC m=+456.548050843"
Sep 16 19:47:05.087958 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:05.087917    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:05.180142 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:05.180055    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:05.916175 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:05.916137    2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cwtqk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 19:47:05.916576 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:05.916211    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk" podUID="e4906dfd-32df-4809-bab9-496d5dc17c2f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:47:08.915761 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:08.915730    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:47:09.284711 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.284622    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:09.401562 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.401528    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:09.925711 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.925677    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"]
Sep 16 19:47:09.929421 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.929382    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.933776 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:47:09.933744    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"openshift-ai-inference-openshift-default-dockercfg-kbzds\" is forbidden: User \"system:node:ip-10-0-141-26.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-26.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-kbzds\"" type="*v1.Secret"
Sep 16 19:47:09.934030 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:47:09.934002    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" is forbidden: User \"system:node:ip-10-0-141-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-26.ec2.internal' and this object" podUID="2097f2b3-4c75-432b-9854-5d745a938aa1" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.956269 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.956172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.956388 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.956297    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.956388 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.956352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khnqq\" (UniqueName: \"kubernetes.io/projected/2097f2b3-4c75-432b-9854-5d745a938aa1-kube-api-access-khnqq\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.956492 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.956439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.956492 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.956470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.956492 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.956485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.956662 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.956540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.956662 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.956592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.956662 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.956639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2097f2b3-4c75-432b-9854-5d745a938aa1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:09.959600 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.959576    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"]
Sep 16 19:47:09.984456 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:09.984430    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:10.057595 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.057556    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.057767 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.057607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.057767 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.057728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.057884 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.057779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.057884 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.057838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.057993 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.057894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2097f2b3-4c75-432b-9854-5d745a938aa1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.057993 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.057926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.057993 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.057957    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.058144 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.057980    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.058144 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.058048    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.058251 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.058122    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khnqq\" (UniqueName: \"kubernetes.io/projected/2097f2b3-4c75-432b-9854-5d745a938aa1-kube-api-access-khnqq\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.058305 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.058290    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.058358 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.058310    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.058713 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.058678    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2097f2b3-4c75-432b-9854-5d745a938aa1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.060174 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.060151    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.060481 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.060460    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.073200 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.073179    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2097f2b3-4c75-432b-9854-5d745a938aa1-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.079343 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.079324    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khnqq\" (UniqueName: \"kubernetes.io/projected/2097f2b3-4c75-432b-9854-5d745a938aa1-kube-api-access-khnqq\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-z4fth\" (UID: \"2097f2b3-4c75-432b-9854-5d745a938aa1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:10.102996 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:10.102969    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:11.025690 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:11.025659    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-kbzds\""
Sep 16 19:47:11.031555 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:11.031532    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:11.309015 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:11.308988    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"]
Sep 16 19:47:11.951119 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:11.951086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" event={"ID":"2097f2b3-4c75-432b-9854-5d745a938aa1","Type":"ContainerStarted","Data":"89120cbac85c0f25c2bec65513b3e38844464bc079438d75a7ec6c5797deb9ad"}
Sep 16 19:47:12.823063 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:12.823020    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:13.808143 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:13.808110    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:47:13.808216 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:13.808182    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:47:13.808216 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:13.808209    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:47:13.964242 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:13.964193    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" event={"ID":"2097f2b3-4c75-432b-9854-5d745a938aa1","Type":"ContainerStarted","Data":"134fe5b11ddf6bf88021b7f6c964892af1b08f77e406b214e3510608510905ef"}
Sep 16 19:47:13.986763 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:13.986710    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" podStartSLOduration=2.494140671 podStartE2EDuration="4.986694873s" totalImagesPullingTime="2.492554202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:13.98435738 +0000 UTC m=+465.541702478" watchObservedRunningTime="2026-09-16 19:47:13.986694873 +0000 UTC m=+465.544039959"
Sep 16 19:47:14.032581 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:14.032544    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:14.034124 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:14.034097    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 19:47:14.034231 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:14.034144    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" podUID="2097f2b3-4c75-432b-9854-5d745a938aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 19:47:15.036041 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:15.036006    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:15.972735 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:15.972703    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:15.973611 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:15.973592    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:47:17.374268 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:17.374230    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:17.521483 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:17.521449    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:20.566110 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.566073    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"]
Sep 16 19:47:20.569657 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.569639    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.572122 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.572101    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 19:47:20.573012 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.572992    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 19:47:20.573098 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.573034    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zzxjc\""
Sep 16 19:47:20.579950 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.579929    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"]
Sep 16 19:47:20.643375 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.643338    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgzk\" (UniqueName: \"kubernetes.io/projected/dc570c56-56b9-4aa3-925e-939de6daa708-kube-api-access-nkgzk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.643601 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.643386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.643601 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.643471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.650548 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.650521    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"]
Sep 16 19:47:20.656785 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.655686    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.664842 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.664806    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"]
Sep 16 19:47:20.744182 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.744144    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"]
Sep 16 19:47:20.744355 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.744285    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgzk\" (UniqueName: \"kubernetes.io/projected/dc570c56-56b9-4aa3-925e-939de6daa708-kube-api-access-nkgzk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.744355 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.744324    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.744506 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.744456    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.744565 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.744500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m25g\" (UniqueName: \"kubernetes.io/projected/f532ad23-f39b-4433-a379-27788f248e15-kube-api-access-9m25g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.744618 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.744589    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.744672 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.744632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.744915 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.744891    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.744987 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.744902    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.747907 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.747887    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.752880 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.752849    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgzk\" (UniqueName: \"kubernetes.io/projected/dc570c56-56b9-4aa3-925e-939de6daa708-kube-api-access-nkgzk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.757812 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.757792    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"]
Sep 16 19:47:20.828241 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.828165    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"]
Sep 16 19:47:20.831728 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.831711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.845169 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.845314 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m25g\" (UniqueName: \"kubernetes.io/projected/f532ad23-f39b-4433-a379-27788f248e15-kube-api-access-9m25g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.845314 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845219    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q8s9\" (UniqueName: \"kubernetes.io/projected/deeb6d71-1a14-417e-b875-c312a982184b-kube-api-access-9q8s9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.845314 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845246    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.845314 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.845553 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgq98\" (UniqueName: \"kubernetes.io/projected/aaece512-e57a-46c4-a7b4-db3571a71f66-kube-api-access-tgq98\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.845553 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845480    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.845553 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.845553 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.845779 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845616    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.845825 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.845790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.847166 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.847146    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"]
Sep 16 19:47:20.859388 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.859354    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m25g\" (UniqueName: \"kubernetes.io/projected/f532ad23-f39b-4433-a379-27788f248e15-kube-api-access-9m25g\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:20.879104 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.879071    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:20.946116 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.946238 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q8s9\" (UniqueName: \"kubernetes.io/projected/deeb6d71-1a14-417e-b875-c312a982184b-kube-api-access-9q8s9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.946238 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946195    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.946238 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946229    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.946338 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgq98\" (UniqueName: \"kubernetes.io/projected/aaece512-e57a-46c4-a7b4-db3571a71f66-kube-api-access-tgq98\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.946338 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.946762 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946548    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.946762 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946647    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.946762 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946728    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.946762 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.946728    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.962326 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.962257    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q8s9\" (UniqueName: \"kubernetes.io/projected/deeb6d71-1a14-417e-b875-c312a982184b-kube-api-access-9q8s9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:20.963206 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.963161    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgq98\" (UniqueName: \"kubernetes.io/projected/aaece512-e57a-46c4-a7b4-db3571a71f66-kube-api-access-tgq98\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:20.967833 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:20.967813    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:21.064197 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:21.064162    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:21.139936 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:21.139910    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:21.156997 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:21.156972    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"]
Sep 16 19:47:21.160450 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:47:21.160415    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc570c56_56b9_4aa3_925e_939de6daa708.slice/crio-5c82347f472d89729a0c44ac0942048025ee2953fd56b661ab4a3b93ff40391f WatchSource:0}: Error finding container 5c82347f472d89729a0c44ac0942048025ee2953fd56b661ab4a3b93ff40391f: Status 404 returned error can't find the container with id 5c82347f472d89729a0c44ac0942048025ee2953fd56b661ab4a3b93ff40391f
Sep 16 19:47:21.246774 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:21.246742    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"]
Sep 16 19:47:21.248537 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:47:21.248511    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf532ad23_f39b_4433_a379_27788f248e15.slice/crio-f558d9abf062020f5be1477514f4b3463cc7c4bfde1fb0d3f661174a19b57906 WatchSource:0}: Error finding container f558d9abf062020f5be1477514f4b3463cc7c4bfde1fb0d3f661174a19b57906: Status 404 returned error can't find the container with id f558d9abf062020f5be1477514f4b3463cc7c4bfde1fb0d3f661174a19b57906
Sep 16 19:47:21.340470 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:21.340436    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"]
Sep 16 19:47:21.352854 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:47:21.352829    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaece512_e57a_46c4_a7b4_db3571a71f66.slice/crio-4841ad1b8d676d653171ebcf5885420f570a5a6d1a19e78d818d0ae24b18be8f WatchSource:0}: Error finding container 4841ad1b8d676d653171ebcf5885420f570a5a6d1a19e78d818d0ae24b18be8f: Status 404 returned error can't find the container with id 4841ad1b8d676d653171ebcf5885420f570a5a6d1a19e78d818d0ae24b18be8f
Sep 16 19:47:21.422001 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:21.421979    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"]
Sep 16 19:47:21.476880 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:47:21.476848    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeb6d71_1a14_417e_b875_c312a982184b.slice/crio-216f21c306e41abc6ed451a2ea6454b19468b8af9cd90fb758c4db3ac2befc3d WatchSource:0}: Error finding container 216f21c306e41abc6ed451a2ea6454b19468b8af9cd90fb758c4db3ac2befc3d: Status 404 returned error can't find the container with id 216f21c306e41abc6ed451a2ea6454b19468b8af9cd90fb758c4db3ac2befc3d
Sep 16 19:47:22.001936 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.001899    2574 generic.go:356] "Generic (PLEG): container finished" podID="deeb6d71-1a14-417e-b875-c312a982184b" containerID="0a6e4af9ebefc599f3a824d7d9c0890ae6dc04ce333ae2d4c43f5991558a635f" exitCode=0
Sep 16 19:47:22.002368 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.001993    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7" event={"ID":"deeb6d71-1a14-417e-b875-c312a982184b","Type":"ContainerDied","Data":"0a6e4af9ebefc599f3a824d7d9c0890ae6dc04ce333ae2d4c43f5991558a635f"}
Sep 16 19:47:22.002368 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.002040    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7" event={"ID":"deeb6d71-1a14-417e-b875-c312a982184b","Type":"ContainerStarted","Data":"216f21c306e41abc6ed451a2ea6454b19468b8af9cd90fb758c4db3ac2befc3d"}
Sep 16 19:47:22.003721 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.003698    2574 generic.go:356] "Generic (PLEG): container finished" podID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerID="706eaf1e1389e6a8b3f0e256599bcea0acfc566ba0f44232e593b8c81df6390e" exitCode=0
Sep 16 19:47:22.003809 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.003786    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2" event={"ID":"aaece512-e57a-46c4-a7b4-db3571a71f66","Type":"ContainerDied","Data":"706eaf1e1389e6a8b3f0e256599bcea0acfc566ba0f44232e593b8c81df6390e"}
Sep 16 19:47:22.003866 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.003835    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2" event={"ID":"aaece512-e57a-46c4-a7b4-db3571a71f66","Type":"ContainerStarted","Data":"4841ad1b8d676d653171ebcf5885420f570a5a6d1a19e78d818d0ae24b18be8f"}
Sep 16 19:47:22.005473 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.005451    2574 generic.go:356] "Generic (PLEG): container finished" podID="f532ad23-f39b-4433-a379-27788f248e15" containerID="cade202400b4d6bd31570a788027848e9b102ac851a16cfa426a63f993b13ec2" exitCode=0
Sep 16 19:47:22.005575 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.005512    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8" event={"ID":"f532ad23-f39b-4433-a379-27788f248e15","Type":"ContainerDied","Data":"cade202400b4d6bd31570a788027848e9b102ac851a16cfa426a63f993b13ec2"}
Sep 16 19:47:22.005575 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.005566    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8" event={"ID":"f532ad23-f39b-4433-a379-27788f248e15","Type":"ContainerStarted","Data":"f558d9abf062020f5be1477514f4b3463cc7c4bfde1fb0d3f661174a19b57906"}
Sep 16 19:47:22.007237 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.007220    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc570c56-56b9-4aa3-925e-939de6daa708" containerID="f1ec39ee827a697f0ee0b667dd26a5866ff7f293439feb8607daacb4eebcfbe2" exitCode=0
Sep 16 19:47:22.007314 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.007272    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4" event={"ID":"dc570c56-56b9-4aa3-925e-939de6daa708","Type":"ContainerDied","Data":"f1ec39ee827a697f0ee0b667dd26a5866ff7f293439feb8607daacb4eebcfbe2"}
Sep 16 19:47:22.007356 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:22.007311    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4" event={"ID":"dc570c56-56b9-4aa3-925e-939de6daa708","Type":"ContainerStarted","Data":"5c82347f472d89729a0c44ac0942048025ee2953fd56b661ab4a3b93ff40391f"}
Sep 16 19:47:23.014539 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:23.014508    2574 generic.go:356] "Generic (PLEG): container finished" podID="deeb6d71-1a14-417e-b875-c312a982184b" containerID="163d46ea75a5e368ab5f1d40732628256bf3fd4447ba1bb55e79f41eb6c5a073" exitCode=0
Sep 16 19:47:23.014967 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:23.014589    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7" event={"ID":"deeb6d71-1a14-417e-b875-c312a982184b","Type":"ContainerDied","Data":"163d46ea75a5e368ab5f1d40732628256bf3fd4447ba1bb55e79f41eb6c5a073"}
Sep 16 19:47:23.017424 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:23.017376    2574 generic.go:356] "Generic (PLEG): container finished" podID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerID="158d279a2189787abb22ad0876817f33d0cdd92361fabfe1d5ac25d28a092b41" exitCode=0
Sep 16 19:47:23.017525 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:23.017489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2" event={"ID":"aaece512-e57a-46c4-a7b4-db3571a71f66","Type":"ContainerDied","Data":"158d279a2189787abb22ad0876817f33d0cdd92361fabfe1d5ac25d28a092b41"}
Sep 16 19:47:23.019513 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:23.019489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4" event={"ID":"dc570c56-56b9-4aa3-925e-939de6daa708","Type":"ContainerStarted","Data":"795b09af1794e86194dc18c7b7a8589df48005217318dce81799a9d8db7f69fe"}
Sep 16 19:47:24.026930 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:24.026846    2574 generic.go:356] "Generic (PLEG): container finished" podID="deeb6d71-1a14-417e-b875-c312a982184b" containerID="3f244212f8a4ad02a199f243322f001132b039c59912b85574b23edde786d4d2" exitCode=0
Sep 16 19:47:24.027324 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:24.026933    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7" event={"ID":"deeb6d71-1a14-417e-b875-c312a982184b","Type":"ContainerDied","Data":"3f244212f8a4ad02a199f243322f001132b039c59912b85574b23edde786d4d2"}
Sep 16 19:47:24.029054 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:24.029033    2574 generic.go:356] "Generic (PLEG): container finished" podID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerID="279ed2215bc7a04b66ff58e6d427fb34fe039ec971375f631015c5dd90db47bd" exitCode=0
Sep 16 19:47:24.029160 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:24.029117    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2" event={"ID":"aaece512-e57a-46c4-a7b4-db3571a71f66","Type":"ContainerDied","Data":"279ed2215bc7a04b66ff58e6d427fb34fe039ec971375f631015c5dd90db47bd"}
Sep 16 19:47:24.031047 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:24.031025    2574 generic.go:356] "Generic (PLEG): container finished" podID="f532ad23-f39b-4433-a379-27788f248e15" containerID="c848fdb9dc31d15cc699eef019ee71a0e16588a673cbbfa75db96dbeec40c72d" exitCode=0
Sep 16 19:47:24.031147 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:24.031115    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8" event={"ID":"f532ad23-f39b-4433-a379-27788f248e15","Type":"ContainerDied","Data":"c848fdb9dc31d15cc699eef019ee71a0e16588a673cbbfa75db96dbeec40c72d"}
Sep 16 19:47:24.032991 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:24.032967    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc570c56-56b9-4aa3-925e-939de6daa708" containerID="795b09af1794e86194dc18c7b7a8589df48005217318dce81799a9d8db7f69fe" exitCode=0
Sep 16 19:47:24.033082 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:24.033047    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4" event={"ID":"dc570c56-56b9-4aa3-925e-939de6daa708","Type":"ContainerDied","Data":"795b09af1794e86194dc18c7b7a8589df48005217318dce81799a9d8db7f69fe"}
Sep 16 19:47:25.040182 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.040151    2574 generic.go:356] "Generic (PLEG): container finished" podID="f532ad23-f39b-4433-a379-27788f248e15" containerID="31b3979cda3e6fae01acae301266c01b4fa4dc846c0f7f6e5c78950802973b18" exitCode=0
Sep 16 19:47:25.040652 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.040235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8" event={"ID":"f532ad23-f39b-4433-a379-27788f248e15","Type":"ContainerDied","Data":"31b3979cda3e6fae01acae301266c01b4fa4dc846c0f7f6e5c78950802973b18"}
Sep 16 19:47:25.042355 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.042333    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc570c56-56b9-4aa3-925e-939de6daa708" containerID="61cce758dc6fb459d7f9965446d0a06cc4baf1622bdf1b243c2d717f68d31818" exitCode=0
Sep 16 19:47:25.042492 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.042431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4" event={"ID":"dc570c56-56b9-4aa3-925e-939de6daa708","Type":"ContainerDied","Data":"61cce758dc6fb459d7f9965446d0a06cc4baf1622bdf1b243c2d717f68d31818"}
Sep 16 19:47:25.191067 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.191041    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:25.194043 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.194023    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:25.285443 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.285395    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-bundle\") pod \"aaece512-e57a-46c4-a7b4-db3571a71f66\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") "
Sep 16 19:47:25.285660 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.285450    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-util\") pod \"aaece512-e57a-46c4-a7b4-db3571a71f66\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") "
Sep 16 19:47:25.285660 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.285489    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q8s9\" (UniqueName: \"kubernetes.io/projected/deeb6d71-1a14-417e-b875-c312a982184b-kube-api-access-9q8s9\") pod \"deeb6d71-1a14-417e-b875-c312a982184b\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") "
Sep 16 19:47:25.285660 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.285504    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgq98\" (UniqueName: \"kubernetes.io/projected/aaece512-e57a-46c4-a7b4-db3571a71f66-kube-api-access-tgq98\") pod \"aaece512-e57a-46c4-a7b4-db3571a71f66\" (UID: \"aaece512-e57a-46c4-a7b4-db3571a71f66\") "
Sep 16 19:47:25.285660 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.285535    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-util\") pod \"deeb6d71-1a14-417e-b875-c312a982184b\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") "
Sep 16 19:47:25.285660 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.285561    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-bundle\") pod \"deeb6d71-1a14-417e-b875-c312a982184b\" (UID: \"deeb6d71-1a14-417e-b875-c312a982184b\") "
Sep 16 19:47:25.286354 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.286006    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-bundle" (OuterVolumeSpecName: "bundle") pod "aaece512-e57a-46c4-a7b4-db3571a71f66" (UID: "aaece512-e57a-46c4-a7b4-db3571a71f66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:25.286354 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.286076    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-bundle" (OuterVolumeSpecName: "bundle") pod "deeb6d71-1a14-417e-b875-c312a982184b" (UID: "deeb6d71-1a14-417e-b875-c312a982184b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:25.287562 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.287533    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deeb6d71-1a14-417e-b875-c312a982184b-kube-api-access-9q8s9" (OuterVolumeSpecName: "kube-api-access-9q8s9") pod "deeb6d71-1a14-417e-b875-c312a982184b" (UID: "deeb6d71-1a14-417e-b875-c312a982184b"). InnerVolumeSpecName "kube-api-access-9q8s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:25.287683 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.287662    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaece512-e57a-46c4-a7b4-db3571a71f66-kube-api-access-tgq98" (OuterVolumeSpecName: "kube-api-access-tgq98") pod "aaece512-e57a-46c4-a7b4-db3571a71f66" (UID: "aaece512-e57a-46c4-a7b4-db3571a71f66"). InnerVolumeSpecName "kube-api-access-tgq98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:25.291007 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.290949    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-util" (OuterVolumeSpecName: "util") pod "deeb6d71-1a14-417e-b875-c312a982184b" (UID: "deeb6d71-1a14-417e-b875-c312a982184b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:25.292539 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.292516    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-util" (OuterVolumeSpecName: "util") pod "aaece512-e57a-46c4-a7b4-db3571a71f66" (UID: "aaece512-e57a-46c4-a7b4-db3571a71f66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:25.386430 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.386381    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q8s9\" (UniqueName: \"kubernetes.io/projected/deeb6d71-1a14-417e-b875-c312a982184b-kube-api-access-9q8s9\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:25.386430 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.386429    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgq98\" (UniqueName: \"kubernetes.io/projected/aaece512-e57a-46c4-a7b4-db3571a71f66-kube-api-access-tgq98\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:25.386619 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.386446    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-util\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:25.386619 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.386459    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/deeb6d71-1a14-417e-b875-c312a982184b-bundle\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:25.386619 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.386467    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-bundle\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:25.386619 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:25.386475    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aaece512-e57a-46c4-a7b4-db3571a71f66-util\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:26.054485 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.054447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7" event={"ID":"deeb6d71-1a14-417e-b875-c312a982184b","Type":"ContainerDied","Data":"216f21c306e41abc6ed451a2ea6454b19468b8af9cd90fb758c4db3ac2befc3d"}
Sep 16 19:47:26.054485 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.054468    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6v6vq7"
Sep 16 19:47:26.054953 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.054529    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="216f21c306e41abc6ed451a2ea6454b19468b8af9cd90fb758c4db3ac2befc3d"
Sep 16 19:47:26.056493 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.056458    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2" event={"ID":"aaece512-e57a-46c4-a7b4-db3571a71f66","Type":"ContainerDied","Data":"4841ad1b8d676d653171ebcf5885420f570a5a6d1a19e78d818d0ae24b18be8f"}
Sep 16 19:47:26.056623 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.056497    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4841ad1b8d676d653171ebcf5885420f570a5a6d1a19e78d818d0ae24b18be8f"
Sep 16 19:47:26.056832 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.056812    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czdgw2"
Sep 16 19:47:26.189013 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.188987    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:26.212489 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.212463    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:26.294949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.294920    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-bundle\") pod \"dc570c56-56b9-4aa3-925e-939de6daa708\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") "
Sep 16 19:47:26.295094 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.294958    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-util\") pod \"dc570c56-56b9-4aa3-925e-939de6daa708\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") "
Sep 16 19:47:26.295094 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.294979    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-bundle\") pod \"f532ad23-f39b-4433-a379-27788f248e15\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") "
Sep 16 19:47:26.295094 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.295028    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m25g\" (UniqueName: \"kubernetes.io/projected/f532ad23-f39b-4433-a379-27788f248e15-kube-api-access-9m25g\") pod \"f532ad23-f39b-4433-a379-27788f248e15\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") "
Sep 16 19:47:26.295094 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.295085    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkgzk\" (UniqueName: \"kubernetes.io/projected/dc570c56-56b9-4aa3-925e-939de6daa708-kube-api-access-nkgzk\") pod \"dc570c56-56b9-4aa3-925e-939de6daa708\" (UID: \"dc570c56-56b9-4aa3-925e-939de6daa708\") "
Sep 16 19:47:26.295334 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.295104    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-util\") pod \"f532ad23-f39b-4433-a379-27788f248e15\" (UID: \"f532ad23-f39b-4433-a379-27788f248e15\") "
Sep 16 19:47:26.295662 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.295631    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-bundle" (OuterVolumeSpecName: "bundle") pod "f532ad23-f39b-4433-a379-27788f248e15" (UID: "f532ad23-f39b-4433-a379-27788f248e15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:26.295827 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.295800    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-bundle" (OuterVolumeSpecName: "bundle") pod "dc570c56-56b9-4aa3-925e-939de6daa708" (UID: "dc570c56-56b9-4aa3-925e-939de6daa708"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:26.297271 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.297240    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f532ad23-f39b-4433-a379-27788f248e15-kube-api-access-9m25g" (OuterVolumeSpecName: "kube-api-access-9m25g") pod "f532ad23-f39b-4433-a379-27788f248e15" (UID: "f532ad23-f39b-4433-a379-27788f248e15"). InnerVolumeSpecName "kube-api-access-9m25g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:26.297358 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.297293    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc570c56-56b9-4aa3-925e-939de6daa708-kube-api-access-nkgzk" (OuterVolumeSpecName: "kube-api-access-nkgzk") pod "dc570c56-56b9-4aa3-925e-939de6daa708" (UID: "dc570c56-56b9-4aa3-925e-939de6daa708"). InnerVolumeSpecName "kube-api-access-nkgzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:26.304019 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.303988    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-util" (OuterVolumeSpecName: "util") pod "dc570c56-56b9-4aa3-925e-939de6daa708" (UID: "dc570c56-56b9-4aa3-925e-939de6daa708"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:26.304104 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.304067    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-util" (OuterVolumeSpecName: "util") pod "f532ad23-f39b-4433-a379-27788f248e15" (UID: "f532ad23-f39b-4433-a379-27788f248e15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:47:26.396439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.396391    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgzk\" (UniqueName: \"kubernetes.io/projected/dc570c56-56b9-4aa3-925e-939de6daa708-kube-api-access-nkgzk\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:26.396439 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.396437    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-util\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:26.396611 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.396449    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-bundle\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:26.396611 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.396457    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc570c56-56b9-4aa3-925e-939de6daa708-util\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:26.396611 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.396464    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f532ad23-f39b-4433-a379-27788f248e15-bundle\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:26.396611 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:26.396472    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m25g\" (UniqueName: \"kubernetes.io/projected/f532ad23-f39b-4433-a379-27788f248e15-kube-api-access-9m25g\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:47:27.063912 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:27.063873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8" event={"ID":"f532ad23-f39b-4433-a379-27788f248e15","Type":"ContainerDied","Data":"f558d9abf062020f5be1477514f4b3463cc7c4bfde1fb0d3f661174a19b57906"}
Sep 16 19:47:27.064376 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:27.063928    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f558d9abf062020f5be1477514f4b3463cc7c4bfde1fb0d3f661174a19b57906"
Sep 16 19:47:27.064376 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:27.063885    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d777fd8"
Sep 16 19:47:27.065832 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:27.065807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4" event={"ID":"dc570c56-56b9-4aa3-925e-939de6daa708","Type":"ContainerDied","Data":"5c82347f472d89729a0c44ac0942048025ee2953fd56b661ab4a3b93ff40391f"}
Sep 16 19:47:27.065832 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:27.065824    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arc6c4"
Sep 16 19:47:27.065987 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:27.065843    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c82347f472d89729a0c44ac0942048025ee2953fd56b661ab4a3b93ff40391f"
Sep 16 19:47:31.059318 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:31.059282    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:31.863634 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:31.863597    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:32.669527 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:32.669494    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.963722    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84fc8d9d69-mx87v"]
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964383    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerName="pull"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964420    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerName="pull"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964439    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc570c56-56b9-4aa3-925e-939de6daa708" containerName="util"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964447    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc570c56-56b9-4aa3-925e-939de6daa708" containerName="util"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964466    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deeb6d71-1a14-417e-b875-c312a982184b" containerName="pull"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964475    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="deeb6d71-1a14-417e-b875-c312a982184b" containerName="pull"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964486    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc570c56-56b9-4aa3-925e-939de6daa708" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964494    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc570c56-56b9-4aa3-925e-939de6daa708" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964521    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerName="util"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964530    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerName="util"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964548    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f532ad23-f39b-4433-a379-27788f248e15" containerName="util"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964556    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f532ad23-f39b-4433-a379-27788f248e15" containerName="util"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964566    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deeb6d71-1a14-417e-b875-c312a982184b" containerName="util"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964574    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="deeb6d71-1a14-417e-b875-c312a982184b" containerName="util"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964593    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc570c56-56b9-4aa3-925e-939de6daa708" containerName="pull"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964601    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc570c56-56b9-4aa3-925e-939de6daa708" containerName="pull"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964618    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f532ad23-f39b-4433-a379-27788f248e15" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964627    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f532ad23-f39b-4433-a379-27788f248e15" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964646    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deeb6d71-1a14-417e-b875-c312a982184b" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964654    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="deeb6d71-1a14-417e-b875-c312a982184b" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964666    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f532ad23-f39b-4433-a379-27788f248e15" containerName="pull"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964677    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f532ad23-f39b-4433-a379-27788f248e15" containerName="pull"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964691    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964700    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964836    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc570c56-56b9-4aa3-925e-939de6daa708" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964855    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f532ad23-f39b-4433-a379-27788f248e15" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964867    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaece512-e57a-46c4-a7b4-db3571a71f66" containerName="extract"
Sep 16 19:47:33.966339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.964882    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="deeb6d71-1a14-417e-b875-c312a982184b" containerName="extract"
Sep 16 19:47:33.971553 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:33.971528    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.000813 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.000789    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84fc8d9d69-mx87v"]
Sep 16 19:47:34.009041 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.009014    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:34.067176 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.067144    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-service-ca\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.067176 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.067176    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-console-config\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.067366 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.067205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-oauth-serving-cert\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.067366 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.067264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4sq\" (UniqueName: \"kubernetes.io/projected/e87100aa-cd82-4865-9f68-2bf2bd1089fd-kube-api-access-ps4sq\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.067459 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.067365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e87100aa-cd82-4865-9f68-2bf2bd1089fd-console-serving-cert\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.067459 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.067434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-trusted-ca-bundle\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.067517 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.067478    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e87100aa-cd82-4865-9f68-2bf2bd1089fd-console-oauth-config\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.168460 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.168397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-service-ca\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.168460 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.168458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-console-config\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.168706 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.168482    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-oauth-serving-cert\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.168706 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.168503    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4sq\" (UniqueName: \"kubernetes.io/projected/e87100aa-cd82-4865-9f68-2bf2bd1089fd-kube-api-access-ps4sq\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.168706 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.168544    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e87100aa-cd82-4865-9f68-2bf2bd1089fd-console-serving-cert\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.168706 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.168677    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-trusted-ca-bundle\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.168902 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.168721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e87100aa-cd82-4865-9f68-2bf2bd1089fd-console-oauth-config\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.169259 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.169234    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-service-ca\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.169259 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.169252    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-console-config\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.169445 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.169258    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-oauth-serving-cert\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.169649 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.169630    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e87100aa-cd82-4865-9f68-2bf2bd1089fd-trusted-ca-bundle\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.171031 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.171014    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e87100aa-cd82-4865-9f68-2bf2bd1089fd-console-serving-cert\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.171146 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.171130    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e87100aa-cd82-4865-9f68-2bf2bd1089fd-console-oauth-config\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.177046 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.177024    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4sq\" (UniqueName: \"kubernetes.io/projected/e87100aa-cd82-4865-9f68-2bf2bd1089fd-kube-api-access-ps4sq\") pod \"console-84fc8d9d69-mx87v\" (UID: \"e87100aa-cd82-4865-9f68-2bf2bd1089fd\") " pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.187213 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.187186    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:34.280189 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.280114    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:34.559509 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.559482    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84fc8d9d69-mx87v"]
Sep 16 19:47:34.560985 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:47:34.560960    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87100aa_cd82_4865_9f68_2bf2bd1089fd.slice/crio-8f71b0074e2cd085fcb32bac969f7407aba990adc7e67417c59a9797ebe5a67b WatchSource:0}: Error finding container 8f71b0074e2cd085fcb32bac969f7407aba990adc7e67417c59a9797ebe5a67b: Status 404 returned error can't find the container with id 8f71b0074e2cd085fcb32bac969f7407aba990adc7e67417c59a9797ebe5a67b
Sep 16 19:47:34.719280 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.719246    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:34.824799 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:34.824720    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:35.105792 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.105756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84fc8d9d69-mx87v" event={"ID":"e87100aa-cd82-4865-9f68-2bf2bd1089fd","Type":"ContainerStarted","Data":"b7d2e4d2bdb6272060d2e2c42008ee5b48329ffd5fe84eaa199ee5d0196683f7"}
Sep 16 19:47:35.106271 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.105820    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84fc8d9d69-mx87v" event={"ID":"e87100aa-cd82-4865-9f68-2bf2bd1089fd","Type":"ContainerStarted","Data":"8f71b0074e2cd085fcb32bac969f7407aba990adc7e67417c59a9797ebe5a67b"}
Sep 16 19:47:35.143492 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.143431    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84fc8d9d69-mx87v" podStartSLOduration=2.14339595 podStartE2EDuration="2.14339595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:35.140355201 +0000 UTC m=+486.697700309" watchObservedRunningTime="2026-09-16 19:47:35.14339595 +0000 UTC m=+486.700741110"
Sep 16 19:47:35.151270 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.151238    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:35.245138 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.245105    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:35.686874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.686841    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"]
Sep 16 19:47:35.690605 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.690585    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:35.696322 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.696206    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 19:47:35.696650 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.696634    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 19:47:35.696951 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.696933    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cmc78\""
Sep 16 19:47:35.714133 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.714106    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"]
Sep 16 19:47:35.784262 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.784225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9514f9df-d0c1-420c-842d-c575c8ddd0a4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vnbr5\" (UID: \"9514f9df-d0c1-420c-842d-c575c8ddd0a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:35.784431 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.784286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbpv6\" (UniqueName: \"kubernetes.io/projected/9514f9df-d0c1-420c-842d-c575c8ddd0a4-kube-api-access-mbpv6\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vnbr5\" (UID: \"9514f9df-d0c1-420c-842d-c575c8ddd0a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:35.884725 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.884688    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9514f9df-d0c1-420c-842d-c575c8ddd0a4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vnbr5\" (UID: \"9514f9df-d0c1-420c-842d-c575c8ddd0a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:35.884866 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.884744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbpv6\" (UniqueName: \"kubernetes.io/projected/9514f9df-d0c1-420c-842d-c575c8ddd0a4-kube-api-access-mbpv6\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vnbr5\" (UID: \"9514f9df-d0c1-420c-842d-c575c8ddd0a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:35.885066 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.885045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9514f9df-d0c1-420c-842d-c575c8ddd0a4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vnbr5\" (UID: \"9514f9df-d0c1-420c-842d-c575c8ddd0a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:35.893587 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:35.893552    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbpv6\" (UniqueName: \"kubernetes.io/projected/9514f9df-d0c1-420c-842d-c575c8ddd0a4-kube-api-access-mbpv6\") pod \"kuadrant-operator-controller-manager-6d4cb79c-vnbr5\" (UID: \"9514f9df-d0c1-420c-842d-c575c8ddd0a4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:36.001187 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:36.001103    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:36.065233 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:36.065205    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:36.291768 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:36.291741    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"]
Sep 16 19:47:36.294615 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:47:36.294590    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9514f9df_d0c1_420c_842d_c575c8ddd0a4.slice/crio-668c2402b66f5ef8b4f07ef9c6a843ce66e0b372c25790faf8382934a3307fe7 WatchSource:0}: Error finding container 668c2402b66f5ef8b4f07ef9c6a843ce66e0b372c25790faf8382934a3307fe7: Status 404 returned error can't find the container with id 668c2402b66f5ef8b4f07ef9c6a843ce66e0b372c25790faf8382934a3307fe7
Sep 16 19:47:37.117478 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.117443    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5" event={"ID":"9514f9df-d0c1-420c-842d-c575c8ddd0a4","Type":"ContainerStarted","Data":"668c2402b66f5ef8b4f07ef9c6a843ce66e0b372c25790faf8382934a3307fe7"}
Sep 16 19:47:37.529272 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.529180    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:37.573167 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.573132    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"]
Sep 16 19:47:37.576651 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.576627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"
Sep 16 19:47:37.581467 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.581444    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vn5dp\""
Sep 16 19:47:37.581605 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.581552    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 19:47:37.595605 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.595584    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"]
Sep 16 19:47:37.702232 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.702199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qx9\" (UniqueName: \"kubernetes.io/projected/e89b4603-bfbd-41be-8d75-27104dedd7cb-kube-api-access-z2qx9\") pod \"dns-operator-controller-manager-65dc8f76f5-bc6tz\" (UID: \"e89b4603-bfbd-41be-8d75-27104dedd7cb\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"
Sep 16 19:47:37.716681 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.716648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:37.803330 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.803248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qx9\" (UniqueName: \"kubernetes.io/projected/e89b4603-bfbd-41be-8d75-27104dedd7cb-kube-api-access-z2qx9\") pod \"dns-operator-controller-manager-65dc8f76f5-bc6tz\" (UID: \"e89b4603-bfbd-41be-8d75-27104dedd7cb\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"
Sep 16 19:47:37.815917 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.815893    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qx9\" (UniqueName: \"kubernetes.io/projected/e89b4603-bfbd-41be-8d75-27104dedd7cb-kube-api-access-z2qx9\") pod \"dns-operator-controller-manager-65dc8f76f5-bc6tz\" (UID: \"e89b4603-bfbd-41be-8d75-27104dedd7cb\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"
Sep 16 19:47:37.890621 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:37.890595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"
Sep 16 19:47:38.207344 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:38.207303    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"]
Sep 16 19:47:38.212846 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:47:38.212812    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89b4603_bfbd_41be_8d75_27104dedd7cb.slice/crio-a1a177836c9a7a6689ac51d2505c19459f62bc8113c48dd2857d26d5742d00a0 WatchSource:0}: Error finding container a1a177836c9a7a6689ac51d2505c19459f62bc8113c48dd2857d26d5742d00a0: Status 404 returned error can't find the container with id a1a177836c9a7a6689ac51d2505c19459f62bc8113c48dd2857d26d5742d00a0
Sep 16 19:47:38.280865 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:38.280820    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:39.131710 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:39.131672    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz" event={"ID":"e89b4603-bfbd-41be-8d75-27104dedd7cb","Type":"ContainerStarted","Data":"a1a177836c9a7a6689ac51d2505c19459f62bc8113c48dd2857d26d5742d00a0"}
Sep 16 19:47:40.185293 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.185250    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"]
Sep 16 19:47:40.189138 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.189110    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"
Sep 16 19:47:40.192455 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.192433    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nrcgc\""
Sep 16 19:47:40.206151 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.206125    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"]
Sep 16 19:47:40.328121 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.328085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hd2z\" (UniqueName: \"kubernetes.io/projected/c725f6be-302d-4e34-a289-c832d7559508-kube-api-access-2hd2z\") pod \"authorino-operator-6fc8f864f4-lxjs5\" (UID: \"c725f6be-302d-4e34-a289-c832d7559508\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"
Sep 16 19:47:40.429617 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.429575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hd2z\" (UniqueName: \"kubernetes.io/projected/c725f6be-302d-4e34-a289-c832d7559508-kube-api-access-2hd2z\") pod \"authorino-operator-6fc8f864f4-lxjs5\" (UID: \"c725f6be-302d-4e34-a289-c832d7559508\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"
Sep 16 19:47:40.446079 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.445997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hd2z\" (UniqueName: \"kubernetes.io/projected/c725f6be-302d-4e34-a289-c832d7559508-kube-api-access-2hd2z\") pod \"authorino-operator-6fc8f864f4-lxjs5\" (UID: \"c725f6be-302d-4e34-a289-c832d7559508\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"
Sep 16 19:47:40.502489 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.502453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"
Sep 16 19:47:40.511821 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.511795    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:40.618943 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.618899    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:40.727035 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:40.726946    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:41.145202 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:41.145165    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5" event={"ID":"9514f9df-d0c1-420c-842d-c575c8ddd0a4","Type":"ContainerStarted","Data":"849a1026ad368e14ce8cb6548d98b66d2c033def977cfbe9aeb015ed11465ab4"}
Sep 16 19:47:41.145396 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:41.145300    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:41.147176 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:41.147151    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz" event={"ID":"e89b4603-bfbd-41be-8d75-27104dedd7cb","Type":"ContainerStarted","Data":"7bb07f4de577babb0bac9b84023e1356586058ba16b86d6391cd11334cac93b5"}
Sep 16 19:47:41.147320 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:41.147265    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"
Sep 16 19:47:41.174781 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:41.174726    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5" podStartSLOduration=1.448733219 podStartE2EDuration="6.174711527s" totalImagesPullingTime="4.725978308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:41.172257027 +0000 UTC m=+492.729602113" watchObservedRunningTime="2026-09-16 19:47:41.174711527 +0000 UTC m=+492.732056615"
Sep 16 19:47:41.207831 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:41.207778    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz" podStartSLOduration=1.407356044 podStartE2EDuration="4.20776299s" totalImagesPullingTime="2.800406946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:41.206507874 +0000 UTC m=+492.763852978" watchObservedRunningTime="2026-09-16 19:47:41.20776299 +0000 UTC m=+492.765108076"
Sep 16 19:47:41.284464 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:41.284439    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"]
Sep 16 19:47:41.286151 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:47:41.286128    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc725f6be_302d_4e34_a289_c832d7559508.slice/crio-6b14e7e59405f0f04ffc71d5de9ba506f72a6fbb91212050a166f3f3c3d76234 WatchSource:0}: Error finding container 6b14e7e59405f0f04ffc71d5de9ba506f72a6fbb91212050a166f3f3c3d76234: Status 404 returned error can't find the container with id 6b14e7e59405f0f04ffc71d5de9ba506f72a6fbb91212050a166f3f3c3d76234
Sep 16 19:47:42.152939 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:42.152908    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5" event={"ID":"c725f6be-302d-4e34-a289-c832d7559508","Type":"ContainerStarted","Data":"6b14e7e59405f0f04ffc71d5de9ba506f72a6fbb91212050a166f3f3c3d76234"}
Sep 16 19:47:42.345422 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:42.345356    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:43.592933 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:43.592895    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:44.003074 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:44.002990    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:44.280430 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:44.280310    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:44.280430 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:44.280369    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:44.285442 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:44.285424    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:45.171751 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:45.171722    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84fc8d9d69-mx87v"
Sep 16 19:47:45.256568 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:45.256536    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-657c5c55c6-lpqcb"]
Sep 16 19:47:45.395229 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:45.395176    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:45.596213 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:45.596184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:45.692551 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:45.692520    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:46.079584 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:46.079551    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:46.174719 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:46.174686    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5" event={"ID":"c725f6be-302d-4e34-a289-c832d7559508","Type":"ContainerStarted","Data":"fd87ee2035c5f7998f8c289127f368a0358a7521053a2a51c81583ec02322205"}
Sep 16 19:47:46.175103 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:46.174918    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"
Sep 16 19:47:46.185342 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:46.185307    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:46.208074 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:46.207993    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5" podStartSLOduration=1.976177152 podStartE2EDuration="6.207975639s" totalImagesPullingTime="4.231798487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:46.207271788 +0000 UTC m=+497.764616907" watchObservedRunningTime="2026-09-16 19:47:46.207975639 +0000 UTC m=+497.765320726"
Sep 16 19:47:46.504356 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:46.504317    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:46.604419 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:46.604380    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:46.912717 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:46.912675    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:46.994658 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:46.994622    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:52.154860 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:52.154832    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-bc6tz"
Sep 16 19:47:52.155242 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:52.155012    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-vnbr5"
Sep 16 19:47:52.608491 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:52.608462    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:52.726132 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:52.726102    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:57.183159 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:57.183128    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-lxjs5"
Sep 16 19:47:59.592473 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:59.592441    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:59.667670 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:47:59.667643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:00.128964 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:00.128928    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:00.229114 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:00.229081    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:03.931137 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:03.931100    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:04.039373 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:04.039335    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:09.736796 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:09.736762    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:09.844194 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:09.844162    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:10.279354 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.279309    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-657c5c55c6-lpqcb" podUID="a9885af7-462a-44b1-99bd-eaa6187aee67" containerName="console" containerID="cri-o://67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b" gracePeriod=15
Sep 16 19:48:10.337107 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.337074    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:10.394386 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.394353    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:10.537342 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.537286    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:48:10.591019 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.590987    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-serving-cert\") pod \"a9885af7-462a-44b1-99bd-eaa6187aee67\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") "
Sep 16 19:48:10.591019 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591023    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-trusted-ca-bundle\") pod \"a9885af7-462a-44b1-99bd-eaa6187aee67\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") "
Sep 16 19:48:10.591230 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591069    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjjjt\" (UniqueName: \"kubernetes.io/projected/a9885af7-462a-44b1-99bd-eaa6187aee67-kube-api-access-cjjjt\") pod \"a9885af7-462a-44b1-99bd-eaa6187aee67\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") "
Sep 16 19:48:10.591230 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591100    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-oauth-config\") pod \"a9885af7-462a-44b1-99bd-eaa6187aee67\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") "
Sep 16 19:48:10.591230 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591122    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-service-ca\") pod \"a9885af7-462a-44b1-99bd-eaa6187aee67\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") "
Sep 16 19:48:10.591230 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591139    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-console-config\") pod \"a9885af7-462a-44b1-99bd-eaa6187aee67\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") "
Sep 16 19:48:10.591230 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591173    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-oauth-serving-cert\") pod \"a9885af7-462a-44b1-99bd-eaa6187aee67\" (UID: \"a9885af7-462a-44b1-99bd-eaa6187aee67\") "
Sep 16 19:48:10.591525 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591477    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a9885af7-462a-44b1-99bd-eaa6187aee67" (UID: "a9885af7-462a-44b1-99bd-eaa6187aee67"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:48:10.591623 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591590    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-console-config" (OuterVolumeSpecName: "console-config") pod "a9885af7-462a-44b1-99bd-eaa6187aee67" (UID: "a9885af7-462a-44b1-99bd-eaa6187aee67"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:48:10.591623 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591602    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-service-ca" (OuterVolumeSpecName: "service-ca") pod "a9885af7-462a-44b1-99bd-eaa6187aee67" (UID: "a9885af7-462a-44b1-99bd-eaa6187aee67"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:48:10.591908 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.591884    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a9885af7-462a-44b1-99bd-eaa6187aee67" (UID: "a9885af7-462a-44b1-99bd-eaa6187aee67"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:48:10.593323 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.593294    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9885af7-462a-44b1-99bd-eaa6187aee67-kube-api-access-cjjjt" (OuterVolumeSpecName: "kube-api-access-cjjjt") pod "a9885af7-462a-44b1-99bd-eaa6187aee67" (UID: "a9885af7-462a-44b1-99bd-eaa6187aee67"). InnerVolumeSpecName "kube-api-access-cjjjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:10.593323 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.593311    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a9885af7-462a-44b1-99bd-eaa6187aee67" (UID: "a9885af7-462a-44b1-99bd-eaa6187aee67"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:48:10.593475 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.593358    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a9885af7-462a-44b1-99bd-eaa6187aee67" (UID: "a9885af7-462a-44b1-99bd-eaa6187aee67"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:48:10.691969 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.691934    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-oauth-config\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.691969 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.691963    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-service-ca\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.691969 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.691972    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-console-config\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.692192 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.691980    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-oauth-serving-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.692192 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.691989    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9885af7-462a-44b1-99bd-eaa6187aee67-console-serving-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.692192 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.691997    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9885af7-462a-44b1-99bd-eaa6187aee67-trusted-ca-bundle\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.692192 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.692005    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjjjt\" (UniqueName: \"kubernetes.io/projected/a9885af7-462a-44b1-99bd-eaa6187aee67-kube-api-access-cjjjt\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:48:10.921480 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.921437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:10.991132 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:10.991089    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:11.301561 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:11.301469    2574 generic.go:356] "Generic (PLEG): container finished" podID="a9885af7-462a-44b1-99bd-eaa6187aee67" containerID="67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b" exitCode=0
Sep 16 19:48:11.301561 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:11.301509    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657c5c55c6-lpqcb"
Sep 16 19:48:11.301846 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:11.301550    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657c5c55c6-lpqcb" event={"ID":"a9885af7-462a-44b1-99bd-eaa6187aee67","Type":"ContainerDied","Data":"67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b"}
Sep 16 19:48:11.301846 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:11.301604    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657c5c55c6-lpqcb" event={"ID":"a9885af7-462a-44b1-99bd-eaa6187aee67","Type":"ContainerDied","Data":"9e68705973ad15dc121ccd8eba9908896fc5ef68efcd52abfc35d59c3943dfb0"}
Sep 16 19:48:11.301846 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:11.301627    2574 scope.go:117] "RemoveContainer" containerID="67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b"
Sep 16 19:48:11.310018 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:11.310001    2574 scope.go:117] "RemoveContainer" containerID="67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b"
Sep 16 19:48:11.310271 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:48:11.310252    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b\": container with ID starting with 67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b not found: ID does not exist" containerID="67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b"
Sep 16 19:48:11.310315 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:11.310280    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b"} err="failed to get container status \"67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b\": rpc error: code = NotFound desc = could not find container \"67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b\": container with ID starting with 67ab7e86b0dfb439eca60ad99a19ff9c5ebb7bd72f01211dfe4030436b602f9b not found: ID does not exist"
Sep 16 19:48:11.352646 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:11.352621    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-657c5c55c6-lpqcb"]
Sep 16 19:48:11.367461 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:11.367439    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-657c5c55c6-lpqcb"]
Sep 16 19:48:13.088307 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:13.088270    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9885af7-462a-44b1-99bd-eaa6187aee67" path="/var/lib/kubelet/pods/a9885af7-462a-44b1-99bd-eaa6187aee67/volumes"
Sep 16 19:48:18.724309 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:18.724271    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:18.781289 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:18.781252    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:23.945989 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:23.945899    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:24.048933 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:24.048897    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:24.809661 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:24.809618    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:24.930323 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:24.930286    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:24.990291 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:24.990256    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:25.077031 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.076949    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"]
Sep 16 19:48:25.077322 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.077309    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9885af7-462a-44b1-99bd-eaa6187aee67" containerName="console"
Sep 16 19:48:25.077391 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.077323    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9885af7-462a-44b1-99bd-eaa6187aee67" containerName="console"
Sep 16 19:48:25.077391 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.077381    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9885af7-462a-44b1-99bd-eaa6187aee67" containerName="console"
Sep 16 19:48:25.080258 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.080241    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:25.082840 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.082813    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tqg27\""
Sep 16 19:48:25.083132 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.083113    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 19:48:25.092033 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.092013    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"]
Sep 16 19:48:25.123065 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.123037    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"]
Sep 16 19:48:25.222251 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.222218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5acbeed4-2a5e-44ad-8946-8faa44367bf5-config-file\") pod \"limitador-limitador-84994f7ffc-d9l9h\" (UID: \"5acbeed4-2a5e-44ad-8946-8faa44367bf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:25.222554 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.222530    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc5r2\" (UniqueName: \"kubernetes.io/projected/5acbeed4-2a5e-44ad-8946-8faa44367bf5-kube-api-access-vc5r2\") pod \"limitador-limitador-84994f7ffc-d9l9h\" (UID: \"5acbeed4-2a5e-44ad-8946-8faa44367bf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:25.323801 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.323770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc5r2\" (UniqueName: \"kubernetes.io/projected/5acbeed4-2a5e-44ad-8946-8faa44367bf5-kube-api-access-vc5r2\") pod \"limitador-limitador-84994f7ffc-d9l9h\" (UID: \"5acbeed4-2a5e-44ad-8946-8faa44367bf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:25.323974 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.323811    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5acbeed4-2a5e-44ad-8946-8faa44367bf5-config-file\") pod \"limitador-limitador-84994f7ffc-d9l9h\" (UID: \"5acbeed4-2a5e-44ad-8946-8faa44367bf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:25.324446 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.324423    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5acbeed4-2a5e-44ad-8946-8faa44367bf5-config-file\") pod \"limitador-limitador-84994f7ffc-d9l9h\" (UID: \"5acbeed4-2a5e-44ad-8946-8faa44367bf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:25.332939 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.332913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc5r2\" (UniqueName: \"kubernetes.io/projected/5acbeed4-2a5e-44ad-8946-8faa44367bf5-kube-api-access-vc5r2\") pod \"limitador-limitador-84994f7ffc-d9l9h\" (UID: \"5acbeed4-2a5e-44ad-8946-8faa44367bf5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:25.391883 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.391853    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:25.662376 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.662353    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"]
Sep 16 19:48:25.664443 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:48:25.664394    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acbeed4_2a5e_44ad_8946_8faa44367bf5.slice/crio-6ee8db3de6c713b556e897d6d597e7d5f25713b6a3ed0311670801d519b8ca12 WatchSource:0}: Error finding container 6ee8db3de6c713b556e897d6d597e7d5f25713b6a3ed0311670801d519b8ca12: Status 404 returned error can't find the container with id 6ee8db3de6c713b556e897d6d597e7d5f25713b6a3ed0311670801d519b8ca12
Sep 16 19:48:25.666630 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.666599    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:25.765953 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:25.765917    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:26.382253 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:26.382220    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h" event={"ID":"5acbeed4-2a5e-44ad-8946-8faa44367bf5","Type":"ContainerStarted","Data":"6ee8db3de6c713b556e897d6d597e7d5f25713b6a3ed0311670801d519b8ca12"}
Sep 16 19:48:27.006572 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:27.006471    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:27.080521 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:27.080487    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:27.392017 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:27.391981    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h" event={"ID":"5acbeed4-2a5e-44ad-8946-8faa44367bf5","Type":"ContainerStarted","Data":"ae1fc481356e8cf14a8822c3bdf2551ced1d97b7e976c5daa81915ccb1da637a"}
Sep 16 19:48:27.392413 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:27.392040    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:27.416475 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:27.416427    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h" podStartSLOduration=1.337113807 podStartE2EDuration="2.416389876s" totalImagesPullingTime="1.079276069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:27.41438047 +0000 UTC m=+538.971725576" watchObservedRunningTime="2026-09-16 19:48:27.416389876 +0000 UTC m=+538.973734955"
Sep 16 19:48:38.398919 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:38.398884    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-d9l9h"
Sep 16 19:48:44.543383 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:44.543340    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:44.749866 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:44.749825    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:45.192272 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:45.192237    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:52.830933 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:52.830896    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:52.924614 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:48:52.924577    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:09.047575 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.047540    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"]
Sep 16 19:49:09.051576 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.051560    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.082220 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.082192    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"]
Sep 16 19:49:09.115237 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.115199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.115380 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.115254    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.115380 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.115335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.115524 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.115388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.115524 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.115438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.115597 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.115564    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.115632 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.115597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9j4c\" (UniqueName: \"kubernetes.io/projected/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-kube-api-access-p9j4c\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.216585 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.216545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.216585 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.216587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.216835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.216643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.216835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.216683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.216835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.216737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.216835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.216790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.216835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.216817    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9j4c\" (UniqueName: \"kubernetes.io/projected/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-kube-api-access-p9j4c\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.217585 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.217555    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.219127 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.219103    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.219127 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.219115    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.219269 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.219131    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.219269 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.219158    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.228959 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.228932    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.228959 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.228950    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9j4c\" (UniqueName: \"kubernetes.io/projected/899dc0d4-c2d7-464a-ba4e-25bca3222d2b-kube-api-access-p9j4c\") pod \"istiod-openshift-gateway-5677ccb49b-kd9zp\" (UID: \"899dc0d4-c2d7-464a-ba4e-25bca3222d2b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.360983 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.360946    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:09.725101 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.725075    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"]
Sep 16 19:49:09.726959 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:49:09.726933    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899dc0d4_c2d7_464a_ba4e_25bca3222d2b.slice/crio-6e07ffe0885ece0a2b3d9de92a7dca4ead3ea5c0210e8d16557c7b1f95a61145 WatchSource:0}: Error finding container 6e07ffe0885ece0a2b3d9de92a7dca4ead3ea5c0210e8d16557c7b1f95a61145: Status 404 returned error can't find the container with id 6e07ffe0885ece0a2b3d9de92a7dca4ead3ea5c0210e8d16557c7b1f95a61145
Sep 16 19:49:09.734329 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.734295    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:49:09.734474 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:09.734355    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:49:10.621764 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:10.621735    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp" event={"ID":"899dc0d4-c2d7-464a-ba4e-25bca3222d2b","Type":"ContainerStarted","Data":"eee60d3c33fabab71cd9fafd68b01ca1db1be6378a691cb7274cf674e99bc962"}
Sep 16 19:49:10.622196 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:10.621779    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp" event={"ID":"899dc0d4-c2d7-464a-ba4e-25bca3222d2b","Type":"ContainerStarted","Data":"6e07ffe0885ece0a2b3d9de92a7dca4ead3ea5c0210e8d16557c7b1f95a61145"}
Sep 16 19:49:10.622196 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:10.621893    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:10.623751 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:10.623724    2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-kd9zp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 19:49:10.623864 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:10.623778    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp" podUID="899dc0d4-c2d7-464a-ba4e-25bca3222d2b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:49:10.695717 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:10.695671    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp" podStartSLOduration=1.695654576 podStartE2EDuration="1.695654576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:10.691345225 +0000 UTC m=+582.248690334" watchObservedRunningTime="2026-09-16 19:49:10.695654576 +0000 UTC m=+582.252999662"
Sep 16 19:49:11.627713 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:11.627686    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-kd9zp"
Sep 16 19:49:11.709232 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:11.709201    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"]
Sep 16 19:49:11.709435 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:11.709395    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk" podUID="e4906dfd-32df-4809-bab9-496d5dc17c2f" containerName="discovery" containerID="cri-o://b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852" gracePeriod=30
Sep 16 19:49:11.962515 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:11.962489    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:49:12.042325 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.042289    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906dfd-32df-4809-bab9-496d5dc17c2f-local-certs\") pod \"e4906dfd-32df-4809-bab9-496d5dc17c2f\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") "
Sep 16 19:49:12.042508 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.042351    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-ca-configmap\") pod \"e4906dfd-32df-4809-bab9-496d5dc17c2f\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") "
Sep 16 19:49:12.042508 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.042394    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lk8j\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-kube-api-access-6lk8j\") pod \"e4906dfd-32df-4809-bab9-496d5dc17c2f\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") "
Sep 16 19:49:12.042508 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.042459    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-cacerts\") pod \"e4906dfd-32df-4809-bab9-496d5dc17c2f\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") "
Sep 16 19:49:12.042508 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.042494    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-token\") pod \"e4906dfd-32df-4809-bab9-496d5dc17c2f\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") "
Sep 16 19:49:12.042715 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.042525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-dns-cert\") pod \"e4906dfd-32df-4809-bab9-496d5dc17c2f\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") "
Sep 16 19:49:12.042715 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.042565    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-kubeconfig\") pod \"e4906dfd-32df-4809-bab9-496d5dc17c2f\" (UID: \"e4906dfd-32df-4809-bab9-496d5dc17c2f\") "
Sep 16 19:49:12.042826 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.042746    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e4906dfd-32df-4809-bab9-496d5dc17c2f" (UID: "e4906dfd-32df-4809-bab9-496d5dc17c2f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:49:12.042889 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.042871    2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-ca-configmap\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:49:12.044783 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.044724    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4906dfd-32df-4809-bab9-496d5dc17c2f-local-certs" (OuterVolumeSpecName: "local-certs") pod "e4906dfd-32df-4809-bab9-496d5dc17c2f" (UID: "e4906dfd-32df-4809-bab9-496d5dc17c2f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:12.044930 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.044841    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e4906dfd-32df-4809-bab9-496d5dc17c2f" (UID: "e4906dfd-32df-4809-bab9-496d5dc17c2f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:12.045005 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.044989    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-cacerts" (OuterVolumeSpecName: "cacerts") pod "e4906dfd-32df-4809-bab9-496d5dc17c2f" (UID: "e4906dfd-32df-4809-bab9-496d5dc17c2f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:12.045101 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.045083    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-kube-api-access-6lk8j" (OuterVolumeSpecName: "kube-api-access-6lk8j") pod "e4906dfd-32df-4809-bab9-496d5dc17c2f" (UID: "e4906dfd-32df-4809-bab9-496d5dc17c2f"). InnerVolumeSpecName "kube-api-access-6lk8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:12.045183 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.045154    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4906dfd-32df-4809-bab9-496d5dc17c2f" (UID: "e4906dfd-32df-4809-bab9-496d5dc17c2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:12.045236 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.045160    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e4906dfd-32df-4809-bab9-496d5dc17c2f" (UID: "e4906dfd-32df-4809-bab9-496d5dc17c2f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:12.143718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.143672    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lk8j\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-kube-api-access-6lk8j\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:49:12.143718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.143707    2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-cacerts\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:49:12.143718 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.143722    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-token\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:49:12.143978 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.143734    2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-csr-dns-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:49:12.143978 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.143746    2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4906dfd-32df-4809-bab9-496d5dc17c2f-istio-kubeconfig\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:49:12.143978 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.143757    2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4906dfd-32df-4809-bab9-496d5dc17c2f-local-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:49:12.632543 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.632510    2574 generic.go:356] "Generic (PLEG): container finished" podID="e4906dfd-32df-4809-bab9-496d5dc17c2f" containerID="b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852" exitCode=0
Sep 16 19:49:12.632929 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.632555    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"
Sep 16 19:49:12.632929 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.632590    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk" event={"ID":"e4906dfd-32df-4809-bab9-496d5dc17c2f","Type":"ContainerDied","Data":"b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852"}
Sep 16 19:49:12.632929 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.632634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk" event={"ID":"e4906dfd-32df-4809-bab9-496d5dc17c2f","Type":"ContainerDied","Data":"914775c49f38354f5b8f22f825ad1259a73d399a178b669e9acae8497e33dec2"}
Sep 16 19:49:12.632929 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.632653    2574 scope.go:117] "RemoveContainer" containerID="b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852"
Sep 16 19:49:12.641681 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.641665    2574 scope.go:117] "RemoveContainer" containerID="b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852"
Sep 16 19:49:12.641956 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:49:12.641940    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852\": container with ID starting with b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852 not found: ID does not exist" containerID="b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852"
Sep 16 19:49:12.642000 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.641966    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852"} err="failed to get container status \"b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852\": rpc error: code = NotFound desc = could not find container \"b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852\": container with ID starting with b8785e50ace7d308b88de2573b43e9a91db8e194b58f99705fa1bd69f1168852 not found: ID does not exist"
Sep 16 19:49:12.666470 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.666447    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"]
Sep 16 19:49:12.669430 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:12.669395    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk"]
Sep 16 19:49:13.088622 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:13.088588    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4906dfd-32df-4809-bab9-496d5dc17c2f" path="/var/lib/kubelet/pods/e4906dfd-32df-4809-bab9-496d5dc17c2f/volumes"
Sep 16 19:49:13.646208 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:13.646174    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:16.915680 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:16.915638    2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cwtqk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 19:49:16.916150 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:16.915739    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cwtqk" podUID="e4906dfd-32df-4809-bab9-496d5dc17c2f" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.29:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:49:24.510277 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:24.510210    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:24.582380 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:24.582337    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:24.699864 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:24.699826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:24.925195 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:24.925157    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:25.029044 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:25.028999    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:25.232918 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:25.232796    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:25.334690 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:25.334613    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:25.389689 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:25.389654    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:28.863020 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:28.862995    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:49:28.866592 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:28.866566    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:49:51.611079 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:49:51.610989    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:43.376647 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:50:43.371846    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:43.436486 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:50:43.436451    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:43.496874 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:50:43.496826    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:43.634418 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:50:43.634314    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:43.801052 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:50:43.801020    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:44.517872 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:50:44.517836    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:44.588363 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:50:44.588324    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:55.474004 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:50:55.473964    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:28.460726 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:51:28.460643    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:18.962061 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:18.962024    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:20.170814 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:20.170779    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:31.226711 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:31.226670    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:31.319437 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:31.319392    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:39.020089 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.020047    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:39.241324 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.241294    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:39.373299 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.373267    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:39.463196 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.463161    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:52:39.463534 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.463522    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4906dfd-32df-4809-bab9-496d5dc17c2f" containerName="discovery"
Sep 16 19:52:39.463596 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.463536    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4906dfd-32df-4809-bab9-496d5dc17c2f" containerName="discovery"
Sep 16 19:52:39.463640 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.463604    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4906dfd-32df-4809-bab9-496d5dc17c2f" containerName="discovery"
Sep 16 19:52:39.466835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.466818    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.469133 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.469111    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 19:52:39.469617 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.469601    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 19:52:39.470166 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.470144    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:52:39.470261 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.470176    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 19:52:39.470261 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.470222    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 19:52:39.470397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.470307    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-b9x46\""
Sep 16 19:52:39.470568 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.470548    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-jrdbq\""
Sep 16 19:52:39.477131 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.477108    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:52:39.517438 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.517389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.517633 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.517462    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9474228-a328-499a-808f-1d47b8129933-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.517633 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.517511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vwgn\" (UniqueName: \"kubernetes.io/projected/d9474228-a328-499a-808f-1d47b8129933-kube-api-access-9vwgn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.517633 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.517541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.517633 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.517570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.517835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.517638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.517835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.517675    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.517835 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.517706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9474228-a328-499a-808f-1d47b8129933-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.618701 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.618666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.618701 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.618704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.618957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.618735    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.618957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.618758    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.618957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.618776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9474228-a328-499a-808f-1d47b8129933-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.618957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.618803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.618957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.618835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9474228-a328-499a-808f-1d47b8129933-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.618957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.618871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vwgn\" (UniqueName: \"kubernetes.io/projected/d9474228-a328-499a-808f-1d47b8129933-kube-api-access-9vwgn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.619290 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.619045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.619290 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.619091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.619290 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.619272    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.619483 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.619333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9474228-a328-499a-808f-1d47b8129933-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.619483 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.619448    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.620927 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.620908    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.621134 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.621115    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9474228-a328-499a-808f-1d47b8129933-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.627137 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.627068    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vwgn\" (UniqueName: \"kubernetes.io/projected/d9474228-a328-499a-808f-1d47b8129933-kube-api-access-9vwgn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:39.682536 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.682496    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:39.778690 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:39.778655    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:52:40.062019 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:40.061987    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:52:40.063350 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:52:40.063318    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9474228_a328_499a_808f_1d47b8129933.slice/crio-4fd1d6814fec702eda059da85c969589f5666143c6c05020ae6469660f82c6cd WatchSource:0}: Error finding container 4fd1d6814fec702eda059da85c969589f5666143c6c05020ae6469660f82c6cd: Status 404 returned error can't find the container with id 4fd1d6814fec702eda059da85c969589f5666143c6c05020ae6469660f82c6cd
Sep 16 19:52:40.065570 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:40.065554    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:52:40.686582 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:40.686528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d9474228-a328-499a-808f-1d47b8129933","Type":"ContainerStarted","Data":"4fd1d6814fec702eda059da85c969589f5666143c6c05020ae6469660f82c6cd"}
Sep 16 19:52:44.716070 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:44.716031    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d9474228-a328-499a-808f-1d47b8129933","Type":"ContainerStarted","Data":"6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29"}
Sep 16 19:52:47.733788 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:47.733751    2574 generic.go:356] "Generic (PLEG): container finished" podID="d9474228-a328-499a-808f-1d47b8129933" containerID="6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29" exitCode=0
Sep 16 19:52:47.734292 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:47.733825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d9474228-a328-499a-808f-1d47b8129933","Type":"ContainerDied","Data":"6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29"}
Sep 16 19:52:51.768971 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:51.768806    2574 generic.go:356] "Generic (PLEG): container finished" podID="2097f2b3-4c75-432b-9854-5d745a938aa1" containerID="134fe5b11ddf6bf88021b7f6c964892af1b08f77e406b214e3510608510905ef" exitCode=0
Sep 16 19:52:51.768971 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:51.768881    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" event={"ID":"2097f2b3-4c75-432b-9854-5d745a938aa1","Type":"ContainerDied","Data":"134fe5b11ddf6bf88021b7f6c964892af1b08f77e406b214e3510608510905ef"}
Sep 16 19:52:51.769555 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:51.769387    2574 scope.go:117] "RemoveContainer" containerID="134fe5b11ddf6bf88021b7f6c964892af1b08f77e406b214e3510608510905ef"
Sep 16 19:52:51.770685 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:51.770657    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:52:51.770814 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:51.770741    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:52:51.770814 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:51.770787    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:52:52.779440 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:52.779389    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" event={"ID":"2097f2b3-4c75-432b-9854-5d745a938aa1","Type":"ContainerStarted","Data":"7b9eb94f705942f3933933e6ffdf8ec80f13c2d32e4c9931b9e85a8b7628a5d5"}
Sep 16 19:52:52.886751 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:52.886606    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:53.032274 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:53.032185    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:52:53.032588 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:53.032559    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 19:52:53.032699 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:53.032623    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" podUID="2097f2b3-4c75-432b-9854-5d745a938aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 19:52:53.231897 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:53.231785    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:54.032867 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:54.032830    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 19:52:54.033345 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:54.032935    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" podUID="2097f2b3-4c75-432b-9854-5d745a938aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 19:52:55.032260 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:55.032219    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 19:52:55.032499 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:55.032290    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" podUID="2097f2b3-4c75-432b-9854-5d745a938aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 19:52:56.032946 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:56.032911    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 19:52:56.033452 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:56.032972    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" podUID="2097f2b3-4c75-432b-9854-5d745a938aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 19:52:57.032207 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:57.032165    2574 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 16 19:52:57.032430 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:57.032232    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth" podUID="2097f2b3-4c75-432b-9854-5d745a938aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 16 19:52:58.745601 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:58.745561    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:52:58.746604 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:58.746419    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:52:58.748066 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:52:58.748039    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-z4fth"
Sep 16 19:53:09.440146 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:53:09.440107    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:10.980057 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:53:10.980005    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:17.938150 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:53:17.938116    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d9474228-a328-499a-808f-1d47b8129933","Type":"ContainerStarted","Data":"dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6"}
Sep 16 19:53:17.961058 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:53:17.961003    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.9933178910000002 podStartE2EDuration="38.96098326s" totalImagesPullingTime="33.380725614s" totalInitContainerRuntime="2.586939755s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:52:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:53:17.959353722 +0000 UTC m=+829.516698844" watchObservedRunningTime="2026-09-16 19:53:17.96098326 +0000 UTC m=+829.518328348"
Sep 16 19:53:24.810221 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:53:24.810181    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:36.616600 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:53:36.616553    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:36.713168 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:53:36.713123    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:14.301521 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:14.301480    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:26.676319 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:26.676224    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:26.770763 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:26.770731    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:27.368514 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:27.368481    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:27.469115 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:27.469074    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:28.901137 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:28.901107    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:54:28.903491 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:28.903465    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:54:33.028189 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:33.028152    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:38.662332 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:38.662291    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:38.720456 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:38.720422    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:55.508223 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:55.508183    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:55.782577 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:54:55.782498    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:23.596788 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:23.596747    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:30.437564 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:30.437516    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:30.536678 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:30.536641    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:30.677621 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:30.677590    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:35.021571 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:35.021530    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:35.116277 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:35.116240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:46.595367 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:46.595330    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:55:47.681218 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:47.681179    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:47.778575 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:47.778539    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:57.578050 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:55:57.577941    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:01.617042 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:01.616990    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="d9474228-a328-499a-808f-1d47b8129933" containerName="main" containerID="cri-o://dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6" gracePeriod=45
Sep 16 19:56:02.368899 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.368871    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:56:02.460348 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460257    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-home\") pod \"d9474228-a328-499a-808f-1d47b8129933\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") "
Sep 16 19:56:02.460348 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460307    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9474228-a328-499a-808f-1d47b8129933-cabundle-cert\") pod \"d9474228-a328-499a-808f-1d47b8129933\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") "
Sep 16 19:56:02.460348 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460344    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-kserve-provision-location\") pod \"d9474228-a328-499a-808f-1d47b8129933\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") "
Sep 16 19:56:02.460675 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460378    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-model-cache\") pod \"d9474228-a328-499a-808f-1d47b8129933\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") "
Sep 16 19:56:02.460675 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460453    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-tmp-dir\") pod \"d9474228-a328-499a-808f-1d47b8129933\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") "
Sep 16 19:56:02.460675 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460508    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-dshm\") pod \"d9474228-a328-499a-808f-1d47b8129933\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") "
Sep 16 19:56:02.460675 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460545    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9474228-a328-499a-808f-1d47b8129933-tls-certs\") pod \"d9474228-a328-499a-808f-1d47b8129933\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") "
Sep 16 19:56:02.460675 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460586    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vwgn\" (UniqueName: \"kubernetes.io/projected/d9474228-a328-499a-808f-1d47b8129933-kube-api-access-9vwgn\") pod \"d9474228-a328-499a-808f-1d47b8129933\" (UID: \"d9474228-a328-499a-808f-1d47b8129933\") "
Sep 16 19:56:02.460910 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460722    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9474228-a328-499a-808f-1d47b8129933" (UID: "d9474228-a328-499a-808f-1d47b8129933"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:02.460910 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460798    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9474228-a328-499a-808f-1d47b8129933-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9474228-a328-499a-808f-1d47b8129933" (UID: "d9474228-a328-499a-808f-1d47b8129933"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:56:02.460910 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460882    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9474228-a328-499a-808f-1d47b8129933-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:56:02.460910 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.460898    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:56:02.461389 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.461348    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-home" (OuterVolumeSpecName: "home") pod "d9474228-a328-499a-808f-1d47b8129933" (UID: "d9474228-a328-499a-808f-1d47b8129933"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:02.463006 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.462974    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-dshm" (OuterVolumeSpecName: "dshm") pod "d9474228-a328-499a-808f-1d47b8129933" (UID: "d9474228-a328-499a-808f-1d47b8129933"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:02.463102 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.463016    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9474228-a328-499a-808f-1d47b8129933-kube-api-access-9vwgn" (OuterVolumeSpecName: "kube-api-access-9vwgn") pod "d9474228-a328-499a-808f-1d47b8129933" (UID: "d9474228-a328-499a-808f-1d47b8129933"). InnerVolumeSpecName "kube-api-access-9vwgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:02.463102 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.463022    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9474228-a328-499a-808f-1d47b8129933-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9474228-a328-499a-808f-1d47b8129933" (UID: "d9474228-a328-499a-808f-1d47b8129933"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:02.472095 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.472065    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9474228-a328-499a-808f-1d47b8129933" (UID: "d9474228-a328-499a-808f-1d47b8129933"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:02.480598 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.480575    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9474228-a328-499a-808f-1d47b8129933" (UID: "d9474228-a328-499a-808f-1d47b8129933"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:02.562207 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.562166    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:56:02.562381 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.562225    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:56:02.562381 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.562237    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:56:02.562381 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.562245    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9474228-a328-499a-808f-1d47b8129933-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:56:02.562381 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.562254    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vwgn\" (UniqueName: \"kubernetes.io/projected/d9474228-a328-499a-808f-1d47b8129933-kube-api-access-9vwgn\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:56:02.562381 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.562262    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9474228-a328-499a-808f-1d47b8129933-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:56:02.835977 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.835940    2574 generic.go:356] "Generic (PLEG): container finished" podID="d9474228-a328-499a-808f-1d47b8129933" containerID="dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6" exitCode=0
Sep 16 19:56:02.836434 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.835997    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 19:56:02.836434 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.836027    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d9474228-a328-499a-808f-1d47b8129933","Type":"ContainerDied","Data":"dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6"}
Sep 16 19:56:02.836434 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.836086    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"d9474228-a328-499a-808f-1d47b8129933","Type":"ContainerDied","Data":"4fd1d6814fec702eda059da85c969589f5666143c6c05020ae6469660f82c6cd"}
Sep 16 19:56:02.836434 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.836105    2574 scope.go:117] "RemoveContainer" containerID="dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6"
Sep 16 19:56:02.845913 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.845897    2574 scope.go:117] "RemoveContainer" containerID="6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29"
Sep 16 19:56:02.853555 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.853536    2574 scope.go:117] "RemoveContainer" containerID="dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6"
Sep 16 19:56:02.853821 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:56:02.853800    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6\": container with ID starting with dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6 not found: ID does not exist" containerID="dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6"
Sep 16 19:56:02.853895 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.853833    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6"} err="failed to get container status \"dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6\": rpc error: code = NotFound desc = could not find container \"dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6\": container with ID starting with dd5fcafaa2d5c1552f9e639ce181194edcf356c64e1106ed15f796e3f0f45fc6 not found: ID does not exist"
Sep 16 19:56:02.853895 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.853859    2574 scope.go:117] "RemoveContainer" containerID="6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29"
Sep 16 19:56:02.854082 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:56:02.854063    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29\": container with ID starting with 6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29 not found: ID does not exist" containerID="6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29"
Sep 16 19:56:02.854128 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.854087    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29"} err="failed to get container status \"6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29\": rpc error: code = NotFound desc = could not find container \"6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29\": container with ID starting with 6634615a5f8c044bdd2b1d155a6325cf3377764a85b3f785d7f2aac5434b7e29 not found: ID does not exist"
Sep 16 19:56:02.858276 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.858257    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:56:02.862125 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:02.862097    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 19:56:03.088779 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:03.088685    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9474228-a328-499a-808f-1d47b8129933" path="/var/lib/kubelet/pods/d9474228-a328-499a-808f-1d47b8129933/volumes"
Sep 16 19:56:20.609676 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:20.609639    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:20.704752 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:20.704709    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:25.066111 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.066059    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:25.163894 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.163855    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:25.279547 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.279514    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:25.404029 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.403995    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"]
Sep 16 19:56:25.404626 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.404599    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9474228-a328-499a-808f-1d47b8129933" containerName="storage-initializer"
Sep 16 19:56:25.404773 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.404631    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9474228-a328-499a-808f-1d47b8129933" containerName="storage-initializer"
Sep 16 19:56:25.404773 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.404671    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9474228-a328-499a-808f-1d47b8129933" containerName="main"
Sep 16 19:56:25.404773 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.404680    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9474228-a328-499a-808f-1d47b8129933" containerName="main"
Sep 16 19:56:25.404949 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.404796    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9474228-a328-499a-808f-1d47b8129933" containerName="main"
Sep 16 19:56:25.410162 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.410138    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.414471 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.414432    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 19:56:25.414754 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.414731    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-rxsb2\""
Sep 16 19:56:25.415225 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.414515    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 19:56:25.417480 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.414505    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:56:25.426484 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.426455    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"]
Sep 16 19:56:25.435043 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.435017    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:25.536244 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.536204    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:25.556424 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.556376    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-credential-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.556608 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.556430    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skntg\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-kube-api-access-skntg\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.556608 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.556482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-istio-token\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.556608 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.556510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-data\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.556608 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.556545    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.556608 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.556560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7a9913-759d-4603-92cc-37e350434aed-istio-podinfo\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.556803 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.556645    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-certs\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.556803 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.556693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-envoy\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.556803 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.556773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7a9913-759d-4603-92cc-37e350434aed-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.657898 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.657794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7a9913-759d-4603-92cc-37e350434aed-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.657898 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.657855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-credential-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.657898 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.657879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skntg\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-kube-api-access-skntg\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658185 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.657911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-istio-token\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658185 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.657935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-data\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658185 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.657969    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658185 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.658100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7a9913-759d-4603-92cc-37e350434aed-istio-podinfo\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658426 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.658180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-certs\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658426 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.658338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-envoy\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658527 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.658352    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-data\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658527 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.658435    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658527 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.658490    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-credential-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658625 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.658538    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-certs\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.658660 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.658642    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7a9913-759d-4603-92cc-37e350434aed-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.660569 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.660541    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-envoy\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.660831 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.660808    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7a9913-759d-4603-92cc-37e350434aed-istio-podinfo\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.676106 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.676080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-istio-token\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.676537 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.676520    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skntg\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-kube-api-access-skntg\") pod \"router-gateway-1-openshift-default-858fc4dc48-kpcgw\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:25.725844 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:25.725816    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:26.008365 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:26.008336    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"]
Sep 16 19:56:26.010351 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:56:26.010327    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7a9913_759d_4603_92cc_37e350434aed.slice/crio-f29ace55c779248df31427932fda1c482a57f622b5b206fafab59fcb038a75f3 WatchSource:0}: Error finding container f29ace55c779248df31427932fda1c482a57f622b5b206fafab59fcb038a75f3: Status 404 returned error can't find the container with id f29ace55c779248df31427932fda1c482a57f622b5b206fafab59fcb038a75f3
Sep 16 19:56:26.012354 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:26.012318    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:56:26.012459 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:26.012415    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:56:26.012534 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:26.012466    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:56:26.960187 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:26.960153    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw" event={"ID":"8b7a9913-759d-4603-92cc-37e350434aed","Type":"ContainerStarted","Data":"03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4"}
Sep 16 19:56:26.960585 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:26.960205    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw" event={"ID":"8b7a9913-759d-4603-92cc-37e350434aed","Type":"ContainerStarted","Data":"f29ace55c779248df31427932fda1c482a57f622b5b206fafab59fcb038a75f3"}
Sep 16 19:56:26.984359 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:26.984309    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw" podStartSLOduration=1.984294164 podStartE2EDuration="1.984294164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:56:26.982644809 +0000 UTC m=+1018.539989913" watchObservedRunningTime="2026-09-16 19:56:26.984294164 +0000 UTC m=+1018.541639250"
Sep 16 19:56:27.726419 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:27.726366    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:27.731610 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:27.731586    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:27.842753 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:27.842718    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:27.966392 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:27.966358    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:27.967473 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:27.967451    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:56:28.774738 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:28.774698    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:31.245831 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:31.245793    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:33.407067 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:33.407031    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:33.692703 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:33.692596    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:33.792768 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:33.792716    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:33.893090 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:33.893049    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:37.066190 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:37.066144    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:37.123709 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:37.123675    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:56.245161 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:56:56.245132    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:05.511903 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:05.511852    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:08.075936 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.075904    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"]
Sep 16 19:57:08.079742 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.079718    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.082719 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.082691    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-7kb2b\""
Sep 16 19:57:08.083978 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.083953    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 16 19:57:08.091994 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.091968    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"]
Sep 16 19:57:08.109487 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.109459    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:08.227136 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.227096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.227313 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.227236    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.227313 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.227297    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gc2\" (UniqueName: \"kubernetes.io/projected/c364d18f-a07a-4562-b4ce-299ca153ccca-kube-api-access-m6gc2\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.227431 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.227342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c364d18f-a07a-4562-b4ce-299ca153ccca-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.227530 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.227510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.227592 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.227557    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.328521 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.328426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.328521 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.328467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.328521 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.328494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gc2\" (UniqueName: \"kubernetes.io/projected/c364d18f-a07a-4562-b4ce-299ca153ccca-kube-api-access-m6gc2\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.328767 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.328617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c364d18f-a07a-4562-b4ce-299ca153ccca-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.328767 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.328689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.328767 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.328731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.328945 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.328925    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.329008 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.328975    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.329155 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.329136    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.330962 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.330938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.331115 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.331098    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c364d18f-a07a-4562-b4ce-299ca153ccca-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.337767 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.337743    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gc2\" (UniqueName: \"kubernetes.io/projected/c364d18f-a07a-4562-b4ce-299ca153ccca-kube-api-access-m6gc2\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.391391 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.391360    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:08.666419 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:08.666356    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"]
Sep 16 19:57:08.667527 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:57:08.667496    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc364d18f_a07a_4562_b4ce_299ca153ccca.slice/crio-7c5fb73900738428dabc071bfbc78a8ced15b122300d7fb5a12ec4246667a63f WatchSource:0}: Error finding container 7c5fb73900738428dabc071bfbc78a8ced15b122300d7fb5a12ec4246667a63f: Status 404 returned error can't find the container with id 7c5fb73900738428dabc071bfbc78a8ced15b122300d7fb5a12ec4246667a63f
Sep 16 19:57:09.190322 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:09.190289    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb" event={"ID":"c364d18f-a07a-4562-b4ce-299ca153ccca","Type":"ContainerStarted","Data":"7c5fb73900738428dabc071bfbc78a8ced15b122300d7fb5a12ec4246667a63f"}
Sep 16 19:57:09.904520 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:09.904482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:10.003783 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.003744    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:10.103505 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.103468    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:10.165872 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.165770    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"]
Sep 16 19:57:10.181664 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.180758    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.184186 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.183634    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 19:57:10.184186 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.183940    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-b8xj5\""
Sep 16 19:57:10.185223 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.184775    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 19:57:10.185223 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.185001    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:57:10.199662 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.199580    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"]
Sep 16 19:57:10.204852 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.204827    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:10.271871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.271827    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:10.371554 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.371519    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-certs\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.371740 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.371560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-data\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.371740 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.371590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-istio-token\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.371740 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.371644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371a49c1-4313-4e87-9105-4f7cc903b448-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.371740 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.371711    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371a49c1-4313-4e87-9105-4f7cc903b448-istio-podinfo\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.371740 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.371736    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-socket\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.371987 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.371760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-credential-socket\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.371987 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.371803    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-envoy\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.371987 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.371841    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5dcc\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-kube-api-access-t5dcc\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.472774 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.472676    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371a49c1-4313-4e87-9105-4f7cc903b448-istio-podinfo\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.472774 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.472739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-socket\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.472774 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.472773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-credential-socket\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473065 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.472810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-envoy\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473065 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.472843    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5dcc\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-kube-api-access-t5dcc\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473065 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.472871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-certs\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473065 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.472892    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-data\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473065 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.472912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-istio-token\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473065 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.472938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371a49c1-4313-4e87-9105-4f7cc903b448-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473382 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.473210    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-socket\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473382 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.473249    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-credential-socket\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473582 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.473549    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-data\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473710 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.473591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-certs\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.473710 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.473657    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371a49c1-4313-4e87-9105-4f7cc903b448-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.475726 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.475673    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-envoy\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.476003 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.475981    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371a49c1-4313-4e87-9105-4f7cc903b448-istio-podinfo\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.481077 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.481056    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-istio-token\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.481397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.481369    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5dcc\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-kube-api-access-t5dcc\") pod \"router-gateway-1-openshift-default-665665857b-lj8zw\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.498177 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.498155    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:10.808910 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.808885    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"]
Sep 16 19:57:10.811021 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:57:10.810996    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371a49c1_4313_4e87_9105_4f7cc903b448.slice/crio-6207a93101f421822b167ec2fdc4cd0d1d1d1f5064c355e2427cb1cbfa0eb985 WatchSource:0}: Error finding container 6207a93101f421822b167ec2fdc4cd0d1d1d1f5064c355e2427cb1cbfa0eb985: Status 404 returned error can't find the container with id 6207a93101f421822b167ec2fdc4cd0d1d1d1f5064c355e2427cb1cbfa0eb985
Sep 16 19:57:10.813273 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.813241    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:57:10.813356 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.813307    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:57:10.813356 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:10.813334    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:57:11.205299 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:11.205257    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb" event={"ID":"c364d18f-a07a-4562-b4ce-299ca153ccca","Type":"ContainerStarted","Data":"43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1"}
Sep 16 19:57:11.206970 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:11.206947    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw" event={"ID":"371a49c1-4313-4e87-9105-4f7cc903b448","Type":"ContainerStarted","Data":"262f5a76651748688e99973c4d0f4e6ac931b4a43ddef7be01ab00d7ad60f6ab"}
Sep 16 19:57:11.207090 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:11.206986    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw" event={"ID":"371a49c1-4313-4e87-9105-4f7cc903b448","Type":"ContainerStarted","Data":"6207a93101f421822b167ec2fdc4cd0d1d1d1f5064c355e2427cb1cbfa0eb985"}
Sep 16 19:57:11.223490 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:11.223444    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb" podStartSLOduration=1.3211777740000001 podStartE2EDuration="3.223429195s" totalImagesPullingTime="1.902251421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:11.221761625 +0000 UTC m=+1062.779106738" watchObservedRunningTime="2026-09-16 19:57:11.223429195 +0000 UTC m=+1062.780774281"
Sep 16 19:57:11.241826 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:11.241776    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw" podStartSLOduration=1.241757286 podStartE2EDuration="1.241757286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:11.238127895 +0000 UTC m=+1062.795473002" watchObservedRunningTime="2026-09-16 19:57:11.241757286 +0000 UTC m=+1062.799102374"
Sep 16 19:57:11.391962 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:11.391930    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:11.499339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:11.499254    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:11.504354 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:11.504326    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:12.212652 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:12.212621    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:12.213558 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:12.213538    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:13.558436 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:13.556878    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:13.878429 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:13.868390    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"]
Sep 16 19:57:13.881058 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:13.881028    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:14.231840    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1" cmd=["/bin/sleep","15"]
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.231888    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb" message="" err=<
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:14.231900    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 19:57:14.231923 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb" podUID="c364d18f-a07a-4562-b4ce-299ca153ccca" containerName="main" containerID="cri-o://43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1"
Sep 16 19:57:14.232422 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.231934    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb" podUID="c364d18f-a07a-4562-b4ce-299ca153ccca" containerName="main" containerID="cri-o://43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1" gracePeriod=60
Sep 16 19:57:14.518022 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.517808    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:14.611723 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.611693    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c364d18f-a07a-4562-b4ce-299ca153ccca-tls-certs\") pod \"c364d18f-a07a-4562-b4ce-299ca153ccca\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") "
Sep 16 19:57:14.611723 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.611726    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-tmp-dir\") pod \"c364d18f-a07a-4562-b4ce-299ca153ccca\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") "
Sep 16 19:57:14.612121 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.611806    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6gc2\" (UniqueName: \"kubernetes.io/projected/c364d18f-a07a-4562-b4ce-299ca153ccca-kube-api-access-m6gc2\") pod \"c364d18f-a07a-4562-b4ce-299ca153ccca\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") "
Sep 16 19:57:14.612121 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.611835    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-model-cache\") pod \"c364d18f-a07a-4562-b4ce-299ca153ccca\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") "
Sep 16 19:57:14.612121 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.611869    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-home\") pod \"c364d18f-a07a-4562-b4ce-299ca153ccca\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") "
Sep 16 19:57:14.612121 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.611929    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-dshm\") pod \"c364d18f-a07a-4562-b4ce-299ca153ccca\" (UID: \"c364d18f-a07a-4562-b4ce-299ca153ccca\") "
Sep 16 19:57:14.612311 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.612121    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c364d18f-a07a-4562-b4ce-299ca153ccca" (UID: "c364d18f-a07a-4562-b4ce-299ca153ccca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:14.612311 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.612192    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-home" (OuterVolumeSpecName: "home") pod "c364d18f-a07a-4562-b4ce-299ca153ccca" (UID: "c364d18f-a07a-4562-b4ce-299ca153ccca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:14.612311 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.612256    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:14.612311 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.612271    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:14.612536 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.612362    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-model-cache" (OuterVolumeSpecName: "model-cache") pod "c364d18f-a07a-4562-b4ce-299ca153ccca" (UID: "c364d18f-a07a-4562-b4ce-299ca153ccca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:14.614107 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.614079    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c364d18f-a07a-4562-b4ce-299ca153ccca-kube-api-access-m6gc2" (OuterVolumeSpecName: "kube-api-access-m6gc2") pod "c364d18f-a07a-4562-b4ce-299ca153ccca" (UID: "c364d18f-a07a-4562-b4ce-299ca153ccca"). InnerVolumeSpecName "kube-api-access-m6gc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:14.614253 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.614117    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-dshm" (OuterVolumeSpecName: "dshm") pod "c364d18f-a07a-4562-b4ce-299ca153ccca" (UID: "c364d18f-a07a-4562-b4ce-299ca153ccca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:14.614253 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.614209    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c364d18f-a07a-4562-b4ce-299ca153ccca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c364d18f-a07a-4562-b4ce-299ca153ccca" (UID: "c364d18f-a07a-4562-b4ce-299ca153ccca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:14.713342 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.713305    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c364d18f-a07a-4562-b4ce-299ca153ccca-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:14.713342 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.713337    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6gc2\" (UniqueName: \"kubernetes.io/projected/c364d18f-a07a-4562-b4ce-299ca153ccca-kube-api-access-m6gc2\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:14.713342 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.713347    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:14.713598 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.713357    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c364d18f-a07a-4562-b4ce-299ca153ccca-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:14.990127 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.990089    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"]
Sep 16 19:57:14.990339 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:14.990310    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw" podUID="8b7a9913-759d-4603-92cc-37e350434aed" containerName="istio-proxy" containerID="cri-o://03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4" gracePeriod=30
Sep 16 19:57:15.042747 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.042706    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:15.230033 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.229990    2574 generic.go:356] "Generic (PLEG): container finished" podID="c364d18f-a07a-4562-b4ce-299ca153ccca" containerID="43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1" exitCode=0
Sep 16 19:57:15.230210 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.230088    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"
Sep 16 19:57:15.230210 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.230175    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb" event={"ID":"c364d18f-a07a-4562-b4ce-299ca153ccca","Type":"ContainerDied","Data":"43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1"}
Sep 16 19:57:15.230332 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.230234    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb" event={"ID":"c364d18f-a07a-4562-b4ce-299ca153ccca","Type":"ContainerDied","Data":"7c5fb73900738428dabc071bfbc78a8ced15b122300d7fb5a12ec4246667a63f"}
Sep 16 19:57:15.230332 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.230262    2574 scope.go:117] "RemoveContainer" containerID="43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1"
Sep 16 19:57:15.240794 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.240505    2574 scope.go:117] "RemoveContainer" containerID="43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1"
Sep 16 19:57:15.240878 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:15.240828    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1\": container with ID starting with 43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1 not found: ID does not exist" containerID="43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1"
Sep 16 19:57:15.240878 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.240863    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1"} err="failed to get container status \"43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1\": rpc error: code = NotFound desc = could not find container \"43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1\": container with ID starting with 43462e5ca20a960be5ae7698542a257de0285b925e3b894ce1cec34382d821f1 not found: ID does not exist"
Sep 16 19:57:15.251387 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.251361    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"]
Sep 16 19:57:15.253430 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:15.253390    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6wgpb"]
Sep 16 19:57:17.089550 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:17.089515    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c364d18f-a07a-4562-b4ce-299ca153ccca" path="/var/lib/kubelet/pods/c364d18f-a07a-4562-b4ce-299ca153ccca/volumes"
Sep 16 19:57:18.247584 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.247559    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:57:18.250741 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.250717    2574 generic.go:356] "Generic (PLEG): container finished" podID="8b7a9913-759d-4603-92cc-37e350434aed" containerID="03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4" exitCode=0
Sep 16 19:57:18.250863 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.250764    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"
Sep 16 19:57:18.250863 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.250798    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw" event={"ID":"8b7a9913-759d-4603-92cc-37e350434aed","Type":"ContainerDied","Data":"03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4"}
Sep 16 19:57:18.250863 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.250852    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw" event={"ID":"8b7a9913-759d-4603-92cc-37e350434aed","Type":"ContainerDied","Data":"f29ace55c779248df31427932fda1c482a57f622b5b206fafab59fcb038a75f3"}
Sep 16 19:57:18.250981 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.250875    2574 scope.go:117] "RemoveContainer" containerID="03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4"
Sep 16 19:57:18.259735 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.259719    2574 scope.go:117] "RemoveContainer" containerID="03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4"
Sep 16 19:57:18.260026 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:18.260008    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4\": container with ID starting with 03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4 not found: ID does not exist" containerID="03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4"
Sep 16 19:57:18.260098 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.260033    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4"} err="failed to get container status \"03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4\": rpc error: code = NotFound desc = could not find container \"03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4\": container with ID starting with 03d5ad287edab12351b7d62c0f4ebcaaa614c119272cb6523309134baeaab8d4 not found: ID does not exist"
Sep 16 19:57:18.346658 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346627    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-istio-token\") pod \"8b7a9913-759d-4603-92cc-37e350434aed\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") "
Sep 16 19:57:18.346819 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346681    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-credential-socket\") pod \"8b7a9913-759d-4603-92cc-37e350434aed\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") "
Sep 16 19:57:18.346819 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346726    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-data\") pod \"8b7a9913-759d-4603-92cc-37e350434aed\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") "
Sep 16 19:57:18.346819 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346752    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-socket\") pod \"8b7a9913-759d-4603-92cc-37e350434aed\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") "
Sep 16 19:57:18.346819 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346780    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7a9913-759d-4603-92cc-37e350434aed-istiod-ca-cert\") pod \"8b7a9913-759d-4603-92cc-37e350434aed\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") "
Sep 16 19:57:18.347040 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346822    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-envoy\") pod \"8b7a9913-759d-4603-92cc-37e350434aed\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") "
Sep 16 19:57:18.347040 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346872    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skntg\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-kube-api-access-skntg\") pod \"8b7a9913-759d-4603-92cc-37e350434aed\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") "
Sep 16 19:57:18.347040 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346914    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b7a9913-759d-4603-92cc-37e350434aed" (UID: "8b7a9913-759d-4603-92cc-37e350434aed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:18.347040 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346921    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-certs\") pod \"8b7a9913-759d-4603-92cc-37e350434aed\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") "
Sep 16 19:57:18.347040 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.346992    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7a9913-759d-4603-92cc-37e350434aed-istio-podinfo\") pod \"8b7a9913-759d-4603-92cc-37e350434aed\" (UID: \"8b7a9913-759d-4603-92cc-37e350434aed\") "
Sep 16 19:57:18.347321 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.347199    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b7a9913-759d-4603-92cc-37e350434aed" (UID: "8b7a9913-759d-4603-92cc-37e350434aed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:18.347321 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.347213    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7a9913-759d-4603-92cc-37e350434aed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b7a9913-759d-4603-92cc-37e350434aed" (UID: "8b7a9913-759d-4603-92cc-37e350434aed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:57:18.347321 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.347293    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b7a9913-759d-4603-92cc-37e350434aed" (UID: "8b7a9913-759d-4603-92cc-37e350434aed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:18.347321 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.347314    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b7a9913-759d-4603-92cc-37e350434aed" (UID: "8b7a9913-759d-4603-92cc-37e350434aed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:18.347556 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.347378    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-credential-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:18.347556 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.347411    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:18.347556 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.347426    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7a9913-759d-4603-92cc-37e350434aed-istiod-ca-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:18.347556 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.347439    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-workload-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:18.349070 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.349044    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b7a9913-759d-4603-92cc-37e350434aed" (UID: "8b7a9913-759d-4603-92cc-37e350434aed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:18.349184 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.349075    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b7a9913-759d-4603-92cc-37e350434aed" (UID: "8b7a9913-759d-4603-92cc-37e350434aed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:18.349184 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.349109    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-kube-api-access-skntg" (OuterVolumeSpecName: "kube-api-access-skntg") pod "8b7a9913-759d-4603-92cc-37e350434aed" (UID: "8b7a9913-759d-4603-92cc-37e350434aed"). InnerVolumeSpecName "kube-api-access-skntg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:18.349184 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.349129    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b7a9913-759d-4603-92cc-37e350434aed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b7a9913-759d-4603-92cc-37e350434aed" (UID: "8b7a9913-759d-4603-92cc-37e350434aed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:57:18.448782 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.448744    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-istio-token\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:18.448782 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.448784    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-data\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:18.449024 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.448797    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7a9913-759d-4603-92cc-37e350434aed-istio-envoy\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:18.449024 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.448812    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skntg\" (UniqueName: \"kubernetes.io/projected/8b7a9913-759d-4603-92cc-37e350434aed-kube-api-access-skntg\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:18.449024 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.448828    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7a9913-759d-4603-92cc-37e350434aed-istio-podinfo\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:18.573551 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.573522    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"]
Sep 16 19:57:18.579068 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:18.579046    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-kpcgw"]
Sep 16 19:57:19.089105 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:19.089072    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7a9913-759d-4603-92cc-37e350434aed" path="/var/lib/kubelet/pods/8b7a9913-759d-4603-92cc-37e350434aed/volumes"
Sep 16 19:57:20.710870 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:20.710834    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:22.202593 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.202554    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:22.603232 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.603201    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"]
Sep 16 19:57:22.603570 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.603556    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7a9913-759d-4603-92cc-37e350434aed" containerName="istio-proxy"
Sep 16 19:57:22.603622 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.603572    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7a9913-759d-4603-92cc-37e350434aed" containerName="istio-proxy"
Sep 16 19:57:22.603622 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.603591    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c364d18f-a07a-4562-b4ce-299ca153ccca" containerName="main"
Sep 16 19:57:22.603622 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.603597    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c364d18f-a07a-4562-b4ce-299ca153ccca" containerName="main"
Sep 16 19:57:22.603712 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.603662    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7a9913-759d-4603-92cc-37e350434aed" containerName="istio-proxy"
Sep 16 19:57:22.603712 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.603675    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c364d18f-a07a-4562-b4ce-299ca153ccca" containerName="main"
Sep 16 19:57:22.610767 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.610747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.614368 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.614343    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-whlcm\""
Sep 16 19:57:22.614512 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.614379    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 16 19:57:22.621262 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.621240    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"]
Sep 16 19:57:22.638822 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.638786    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:22.682617 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.682584    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvlfz\" (UniqueName: \"kubernetes.io/projected/7fc44412-4170-4da1-b6a6-9dd30c81690b-kube-api-access-fvlfz\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.682762 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.682630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.682762 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.682663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.682762 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.682707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.682928 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.682774    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.682928 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.682838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.783541 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.783500    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvlfz\" (UniqueName: \"kubernetes.io/projected/7fc44412-4170-4da1-b6a6-9dd30c81690b-kube-api-access-fvlfz\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.783724 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.783560    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.783724 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.783592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.783724 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.783642    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.783724 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.783682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.783955 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.783727    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.784122 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.784086    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.784213 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.784122    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.784213 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.784145    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.785968 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.785944    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.786452 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.786433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.796323 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.796296    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvlfz\" (UniqueName: \"kubernetes.io/projected/7fc44412-4170-4da1-b6a6-9dd30c81690b-kube-api-access-fvlfz\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:22.923508 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:22.923397    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:23.206135 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:23.206111    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"]
Sep 16 19:57:23.208225 ip-10-0-141-26 kubenswrapper[2574]: W0916 19:57:23.208190    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc44412_4170_4da1_b6a6_9dd30c81690b.slice/crio-94fa689054ea530f593e74f4c197f4ba61e168179d1d27739b61d4db8f959000 WatchSource:0}: Error finding container 94fa689054ea530f593e74f4c197f4ba61e168179d1d27739b61d4db8f959000: Status 404 returned error can't find the container with id 94fa689054ea530f593e74f4c197f4ba61e168179d1d27739b61d4db8f959000
Sep 16 19:57:23.280435 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:23.280391    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" event={"ID":"7fc44412-4170-4da1-b6a6-9dd30c81690b","Type":"ContainerStarted","Data":"94fa689054ea530f593e74f4c197f4ba61e168179d1d27739b61d4db8f959000"}
Sep 16 19:57:24.287477 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:24.287431    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" event={"ID":"7fc44412-4170-4da1-b6a6-9dd30c81690b","Type":"ContainerStarted","Data":"7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7"}
Sep 16 19:57:24.306192 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:24.306142    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" podStartSLOduration=2.306128169 podStartE2EDuration="2.306128169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:24.30435547 +0000 UTC m=+1075.861700586" watchObservedRunningTime="2026-09-16 19:57:24.306128169 +0000 UTC m=+1075.863473256"
Sep 16 19:57:24.621848 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:24.621817    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:24.732365 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:24.732322    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:24.819470 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:24.819442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:24.924279 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:24.924193    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:25.695531 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:25.695496    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"]
Sep 16 19:57:25.713397 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:25.713371    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:26.418389 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:26.418354    2574 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 19:57:26.418610 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:26.418486    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs podName:7fc44412-4170-4da1-b6a6-9dd30c81690b nodeName:}" failed. No retries permitted until 2026-09-16 19:57:26.918464863 +0000 UTC m=+1078.475809931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" (UID: "7fc44412-4170-4da1-b6a6-9dd30c81690b") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 19:57:26.924319 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:26.924280    2574 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 19:57:26.924814 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:26.924366    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs podName:7fc44412-4170-4da1-b6a6-9dd30c81690b nodeName:}" failed. No retries permitted until 2026-09-16 19:57:27.924346727 +0000 UTC m=+1079.481691807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" (UID: "7fc44412-4170-4da1-b6a6-9dd30c81690b") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:27.312436    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7" cmd=["/bin/sleep","15"]
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.312485    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" message="" err=<
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:27.312496    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 19:57:27.312519 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" podUID="7fc44412-4170-4da1-b6a6-9dd30c81690b" containerName="main" containerID="cri-o://7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7"
Sep 16 19:57:27.313130 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.312540    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" podUID="7fc44412-4170-4da1-b6a6-9dd30c81690b" containerName="main" containerID="cri-o://7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7" gracePeriod=60
Sep 16 19:57:27.709645 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.709615    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:27.833448 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833392    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-home\") pod \"7fc44412-4170-4da1-b6a6-9dd30c81690b\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") "
Sep 16 19:57:27.833624 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833461    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvlfz\" (UniqueName: \"kubernetes.io/projected/7fc44412-4170-4da1-b6a6-9dd30c81690b-kube-api-access-fvlfz\") pod \"7fc44412-4170-4da1-b6a6-9dd30c81690b\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") "
Sep 16 19:57:27.833624 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833492    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-tmp-dir\") pod \"7fc44412-4170-4da1-b6a6-9dd30c81690b\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") "
Sep 16 19:57:27.833624 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833550    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-model-cache\") pod \"7fc44412-4170-4da1-b6a6-9dd30c81690b\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") "
Sep 16 19:57:27.833624 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833595    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs\") pod \"7fc44412-4170-4da1-b6a6-9dd30c81690b\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") "
Sep 16 19:57:27.833624 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833619    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-dshm\") pod \"7fc44412-4170-4da1-b6a6-9dd30c81690b\" (UID: \"7fc44412-4170-4da1-b6a6-9dd30c81690b\") "
Sep 16 19:57:27.833889 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833645    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-home" (OuterVolumeSpecName: "home") pod "7fc44412-4170-4da1-b6a6-9dd30c81690b" (UID: "7fc44412-4170-4da1-b6a6-9dd30c81690b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:27.833889 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833808    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fc44412-4170-4da1-b6a6-9dd30c81690b" (UID: "7fc44412-4170-4da1-b6a6-9dd30c81690b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:27.833889 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833829    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fc44412-4170-4da1-b6a6-9dd30c81690b" (UID: "7fc44412-4170-4da1-b6a6-9dd30c81690b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:27.833995 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833919    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:27.833995 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833936    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:27.833995 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.833945    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:27.835818 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.835786    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-dshm" (OuterVolumeSpecName: "dshm") pod "7fc44412-4170-4da1-b6a6-9dd30c81690b" (UID: "7fc44412-4170-4da1-b6a6-9dd30c81690b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:27.835957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.835842    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc44412-4170-4da1-b6a6-9dd30c81690b-kube-api-access-fvlfz" (OuterVolumeSpecName: "kube-api-access-fvlfz") pod "7fc44412-4170-4da1-b6a6-9dd30c81690b" (UID: "7fc44412-4170-4da1-b6a6-9dd30c81690b"). InnerVolumeSpecName "kube-api-access-fvlfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:27.835957 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.835920    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fc44412-4170-4da1-b6a6-9dd30c81690b" (UID: "7fc44412-4170-4da1-b6a6-9dd30c81690b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:27.935223 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.935136    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc44412-4170-4da1-b6a6-9dd30c81690b-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:27.935223 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.935161    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc44412-4170-4da1-b6a6-9dd30c81690b-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:27.935223 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:27.935171    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvlfz\" (UniqueName: \"kubernetes.io/projected/7fc44412-4170-4da1-b6a6-9dd30c81690b-kube-api-access-fvlfz\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:28.311084 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:28.310982    2574 generic.go:356] "Generic (PLEG): container finished" podID="7fc44412-4170-4da1-b6a6-9dd30c81690b" containerID="7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7" exitCode=0
Sep 16 19:57:28.311084 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:28.311036    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"
Sep 16 19:57:28.311084 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:28.311040    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" event={"ID":"7fc44412-4170-4da1-b6a6-9dd30c81690b","Type":"ContainerDied","Data":"7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7"}
Sep 16 19:57:28.311304 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:28.311105    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg" event={"ID":"7fc44412-4170-4da1-b6a6-9dd30c81690b","Type":"ContainerDied","Data":"94fa689054ea530f593e74f4c197f4ba61e168179d1d27739b61d4db8f959000"}
Sep 16 19:57:28.311304 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:28.311138    2574 scope.go:117] "RemoveContainer" containerID="7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7"
Sep 16 19:57:28.320534 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:28.320314    2574 scope.go:117] "RemoveContainer" containerID="7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7"
Sep 16 19:57:28.320651 ip-10-0-141-26 kubenswrapper[2574]: E0916 19:57:28.320630    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7\": container with ID starting with 7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7 not found: ID does not exist" containerID="7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7"
Sep 16 19:57:28.320694 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:28.320661    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7"} err="failed to get container status \"7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7\": rpc error: code = NotFound desc = could not find container \"7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7\": container with ID starting with 7e54e87abf22d28367f599707b427626e88f27d7f0295ca3bd15575a319261f7 not found: ID does not exist"
Sep 16 19:57:28.335889 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:28.335866    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"]
Sep 16 19:57:28.346229 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:28.346201    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvzjfg"]
Sep 16 19:57:29.089552 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:29.089520    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fc44412-4170-4da1-b6a6-9dd30c81690b" path="/var/lib/kubelet/pods/7fc44412-4170-4da1-b6a6-9dd30c81690b/volumes"
Sep 16 19:57:29.600273 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:29.600236    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"]
Sep 16 19:57:29.600494 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:29.600471    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw" podUID="371a49c1-4313-4e87-9105-4f7cc903b448" containerName="istio-proxy" containerID="cri-o://262f5a76651748688e99973c4d0f4e6ac931b4a43ddef7be01ab00d7ad60f6ab" gracePeriod=30
Sep 16 19:57:29.643473 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:29.643354    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:32.177763 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.177721    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:32.333075 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.333045    2574 generic.go:356] "Generic (PLEG): container finished" podID="371a49c1-4313-4e87-9105-4f7cc903b448" containerID="262f5a76651748688e99973c4d0f4e6ac931b4a43ddef7be01ab00d7ad60f6ab" exitCode=0
Sep 16 19:57:32.333247 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.333088    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw" event={"ID":"371a49c1-4313-4e87-9105-4f7cc903b448","Type":"ContainerDied","Data":"262f5a76651748688e99973c4d0f4e6ac931b4a43ddef7be01ab00d7ad60f6ab"}
Sep 16 19:57:32.452045 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.452022    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:32.580092 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580062    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-istio-token\") pod \"371a49c1-4313-4e87-9105-4f7cc903b448\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") "
Sep 16 19:57:32.580283 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580107    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-certs\") pod \"371a49c1-4313-4e87-9105-4f7cc903b448\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") "
Sep 16 19:57:32.580283 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580156    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-socket\") pod \"371a49c1-4313-4e87-9105-4f7cc903b448\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") "
Sep 16 19:57:32.580283 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580175    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-data\") pod \"371a49c1-4313-4e87-9105-4f7cc903b448\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") "
Sep 16 19:57:32.580283 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580198    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-envoy\") pod \"371a49c1-4313-4e87-9105-4f7cc903b448\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") "
Sep 16 19:57:32.580283 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580233    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371a49c1-4313-4e87-9105-4f7cc903b448-istiod-ca-cert\") pod \"371a49c1-4313-4e87-9105-4f7cc903b448\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") "
Sep 16 19:57:32.580283 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580258    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-credential-socket\") pod \"371a49c1-4313-4e87-9105-4f7cc903b448\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") "
Sep 16 19:57:32.580871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580296    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371a49c1-4313-4e87-9105-4f7cc903b448-istio-podinfo\") pod \"371a49c1-4313-4e87-9105-4f7cc903b448\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") "
Sep 16 19:57:32.580871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580322    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5dcc\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-kube-api-access-t5dcc\") pod \"371a49c1-4313-4e87-9105-4f7cc903b448\" (UID: \"371a49c1-4313-4e87-9105-4f7cc903b448\") "
Sep 16 19:57:32.580871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580515    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "371a49c1-4313-4e87-9105-4f7cc903b448" (UID: "371a49c1-4313-4e87-9105-4f7cc903b448"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:32.580871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580527    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "371a49c1-4313-4e87-9105-4f7cc903b448" (UID: "371a49c1-4313-4e87-9105-4f7cc903b448"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:32.580871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580665    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-data" (OuterVolumeSpecName: "istio-data") pod "371a49c1-4313-4e87-9105-4f7cc903b448" (UID: "371a49c1-4313-4e87-9105-4f7cc903b448"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:32.580871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580715    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:32.580871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580728    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "371a49c1-4313-4e87-9105-4f7cc903b448" (UID: "371a49c1-4313-4e87-9105-4f7cc903b448"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:32.580871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580758    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-workload-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:32.581222 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.580906    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371a49c1-4313-4e87-9105-4f7cc903b448-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "371a49c1-4313-4e87-9105-4f7cc903b448" (UID: "371a49c1-4313-4e87-9105-4f7cc903b448"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:57:32.582889 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.582863    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-kube-api-access-t5dcc" (OuterVolumeSpecName: "kube-api-access-t5dcc") pod "371a49c1-4313-4e87-9105-4f7cc903b448" (UID: "371a49c1-4313-4e87-9105-4f7cc903b448"). InnerVolumeSpecName "kube-api-access-t5dcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:32.583110 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.583089    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-istio-token" (OuterVolumeSpecName: "istio-token") pod "371a49c1-4313-4e87-9105-4f7cc903b448" (UID: "371a49c1-4313-4e87-9105-4f7cc903b448"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:32.583203 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.583172    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "371a49c1-4313-4e87-9105-4f7cc903b448" (UID: "371a49c1-4313-4e87-9105-4f7cc903b448"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:32.583450 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.583429    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/371a49c1-4313-4e87-9105-4f7cc903b448-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "371a49c1-4313-4e87-9105-4f7cc903b448" (UID: "371a49c1-4313-4e87-9105-4f7cc903b448"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:57:32.681642 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.681606    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-data\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:32.681642 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.681634    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-istio-envoy\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:32.681642 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.681643    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371a49c1-4313-4e87-9105-4f7cc903b448-istiod-ca-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:32.681872 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.681652    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371a49c1-4313-4e87-9105-4f7cc903b448-credential-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:32.681872 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.681661    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371a49c1-4313-4e87-9105-4f7cc903b448-istio-podinfo\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:32.681872 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.681670    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5dcc\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-kube-api-access-t5dcc\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:32.681872 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:32.681678    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371a49c1-4313-4e87-9105-4f7cc903b448-istio-token\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 19:57:33.340128 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:33.340088    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw" event={"ID":"371a49c1-4313-4e87-9105-4f7cc903b448","Type":"ContainerDied","Data":"6207a93101f421822b167ec2fdc4cd0d1d1d1f5064c355e2427cb1cbfa0eb985"}
Sep 16 19:57:33.340548 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:33.340148    2574 scope.go:117] "RemoveContainer" containerID="262f5a76651748688e99973c4d0f4e6ac931b4a43ddef7be01ab00d7ad60f6ab"
Sep 16 19:57:33.340548 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:33.340099    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"
Sep 16 19:57:33.358607 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:33.358554    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"]
Sep 16 19:57:33.362825 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:33.362799    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-665665857b-lj8zw"]
Sep 16 19:57:35.093045 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:35.088145    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="371a49c1-4313-4e87-9105-4f7cc903b448" path="/var/lib/kubelet/pods/371a49c1-4313-4e87-9105-4f7cc903b448/volumes"
Sep 16 19:57:35.399562 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:35.399530    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:35.938551 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:35.938511    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:36.884380 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:36.884338    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:36.983846 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:36.983808    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:41.020872 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:41.020834    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:41.184915 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:41.184827    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:41.997473 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:41.997435    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:46.100951 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:46.100907    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:51.957312 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:51.957272    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:52.058715 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:57:52.058678    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:14.343299 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:58:14.343259    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:14.428789 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:58:14.428753    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:20.782157 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:58:20.782115    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:26.864811 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:58:26.864770    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:26.922826 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:58:26.922792    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:33.492036 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:58:33.491993    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:33.767661 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:58:33.767576    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:45.430871 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:58:45.430829    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:08.505501 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:08.505418    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:08.637873 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:08.637837    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:08.737192 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:08.737158    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:28.944213 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:28.944183    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:59:28.945058 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:28.945026    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 19:59:38.648783 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:38.648744    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:46.191545 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:46.191509    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:46.307712 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:46.307675    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:57.770848 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:57.770811    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:57.855129 ip-10-0-141-26 kubenswrapper[2574]: I0916 19:59:57.855094    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:04.800494 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:04.800454    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:08.975311 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.975276    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"]
Sep 16 20:00:08.975679 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.975628    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc44412-4170-4da1-b6a6-9dd30c81690b" containerName="main"
Sep 16 20:00:08.975679 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.975639    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc44412-4170-4da1-b6a6-9dd30c81690b" containerName="main"
Sep 16 20:00:08.975679 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.975652    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371a49c1-4313-4e87-9105-4f7cc903b448" containerName="istio-proxy"
Sep 16 20:00:08.975679 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.975657    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="371a49c1-4313-4e87-9105-4f7cc903b448" containerName="istio-proxy"
Sep 16 20:00:08.975815 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.975709    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="371a49c1-4313-4e87-9105-4f7cc903b448" containerName="istio-proxy"
Sep 16 20:00:08.975815 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.975722    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc44412-4170-4da1-b6a6-9dd30c81690b" containerName="main"
Sep 16 20:00:08.978641 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.978626    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:08.984183 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.984159    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 20:00:08.985123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.985097    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-cjbz8\""
Sep 16 20:00:08.985266 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.985180    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 20:00:08.985653 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:08.985463    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 20:00:09.004777 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.004750    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:09.013625 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.013597    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"]
Sep 16 20:00:09.133792 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.133763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.133968 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.133810    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.133968 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.133848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.133968 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.133880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcmf\" (UniqueName: \"kubernetes.io/projected/a73a0efe-90c0-4dfe-a76f-b7298db61e07-kube-api-access-mhcmf\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.133968 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.133911    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.133968 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.133937    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.235337 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.235252    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.235337 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.235288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.235547 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.235355    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.236274 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.236124    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.236274 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.236189    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.236503 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.236295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.236503 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.236366    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.236503 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.236444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcmf\" (UniqueName: \"kubernetes.io/projected/a73a0efe-90c0-4dfe-a76f-b7298db61e07-kube-api-access-mhcmf\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.236954 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.236932    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.242703 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.242678    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.243241 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.243222    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.250455 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.250432    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcmf\" (UniqueName: \"kubernetes.io/projected/a73a0efe-90c0-4dfe-a76f-b7298db61e07-kube-api-access-mhcmf\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.290265 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.290232    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:09.539767 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.539678    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:09.575339 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:09.575202    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"]
Sep 16 20:00:09.577696 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:00:09.577664    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73a0efe_90c0_4dfe_a76f_b7298db61e07.slice/crio-1e6952822c15492a699b9edd42cd1a9b34580c7784b330429fbe35bfe4cb70c0 WatchSource:0}: Error finding container 1e6952822c15492a699b9edd42cd1a9b34580c7784b330429fbe35bfe4cb70c0: Status 404 returned error can't find the container with id 1e6952822c15492a699b9edd42cd1a9b34580c7784b330429fbe35bfe4cb70c0
Sep 16 20:00:10.145088 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:10.145053    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4" event={"ID":"a73a0efe-90c0-4dfe-a76f-b7298db61e07","Type":"ContainerStarted","Data":"260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba"}
Sep 16 20:00:10.145506 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:10.145108    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4" event={"ID":"a73a0efe-90c0-4dfe-a76f-b7298db61e07","Type":"ContainerStarted","Data":"1e6952822c15492a699b9edd42cd1a9b34580c7784b330429fbe35bfe4cb70c0"}
Sep 16 20:00:10.281862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:10.281810    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4" podStartSLOduration=2.2817932 podStartE2EDuration="2.2817932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:00:10.273476063 +0000 UTC m=+1241.830821171" watchObservedRunningTime="2026-09-16 20:00:10.2817932 +0000 UTC m=+1241.839138334"
Sep 16 20:00:10.291286 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:10.291254    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:12.428782 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:12.428738    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:12.517603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:12.517561    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:14.537790 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:14.537755    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:19.291062 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:19.291025    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:19.303749 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:19.303719    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:20.211600 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:20.211569    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:34.909543 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:34.909464    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:36.293301 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:36.293265    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:36.405953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:36.405916    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:43.685689 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:43.685644    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:43.777439 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:43.777386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:44.117386 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:44.117342    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:48.534000 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:48.533965    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:49.967949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:49.967907    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:50.051886 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:50.051848    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:53.871843 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:53.871804    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:53.872378 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:53.871927    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"]
Sep 16 20:00:53.881481 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:00:53.881446    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:00:53.881481 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:00:53.881481 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:00:53.881481 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba" cmd=["/bin/sleep","15"]
Sep 16 20:00:53.881798 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:53.881497    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4" message="" err=<
Sep 16 20:00:53.881798 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:00:53.881798 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:00:53.881798 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:00:53.881798 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:00:53.881509    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:00:53.881798 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:00:53.881798 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:00:53.881798 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4" podUID="a73a0efe-90c0-4dfe-a76f-b7298db61e07" containerName="main" containerID="cri-o://260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba"
Sep 16 20:00:53.881798 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:53.881556    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4" podUID="a73a0efe-90c0-4dfe-a76f-b7298db61e07" containerName="main" containerID="cri-o://260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba" gracePeriod=60
Sep 16 20:00:53.939753 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:53.939704    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:54.225129 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.225106    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:54.332378 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332347    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tmp-dir\") pod \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") "
Sep 16 20:00:54.332378 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332388    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-model-cache\") pod \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") "
Sep 16 20:00:54.332647 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332451    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhcmf\" (UniqueName: \"kubernetes.io/projected/a73a0efe-90c0-4dfe-a76f-b7298db61e07-kube-api-access-mhcmf\") pod \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") "
Sep 16 20:00:54.332647 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332482    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-home\") pod \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") "
Sep 16 20:00:54.332647 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332532    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tls-certs\") pod \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") "
Sep 16 20:00:54.332647 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332560    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-dshm\") pod \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\" (UID: \"a73a0efe-90c0-4dfe-a76f-b7298db61e07\") "
Sep 16 20:00:54.332647 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332616    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a73a0efe-90c0-4dfe-a76f-b7298db61e07" (UID: "a73a0efe-90c0-4dfe-a76f-b7298db61e07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:54.332920 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332753    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-model-cache" (OuterVolumeSpecName: "model-cache") pod "a73a0efe-90c0-4dfe-a76f-b7298db61e07" (UID: "a73a0efe-90c0-4dfe-a76f-b7298db61e07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:54.332920 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332786    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-home" (OuterVolumeSpecName: "home") pod "a73a0efe-90c0-4dfe-a76f-b7298db61e07" (UID: "a73a0efe-90c0-4dfe-a76f-b7298db61e07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:54.332920 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332864    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:00:54.332920 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332880    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:00:54.332920 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.332892    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:00:54.334773 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.334739    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a73a0efe-90c0-4dfe-a76f-b7298db61e07" (UID: "a73a0efe-90c0-4dfe-a76f-b7298db61e07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:54.334773 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.334752    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-dshm" (OuterVolumeSpecName: "dshm") pod "a73a0efe-90c0-4dfe-a76f-b7298db61e07" (UID: "a73a0efe-90c0-4dfe-a76f-b7298db61e07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:54.334893 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.334828    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73a0efe-90c0-4dfe-a76f-b7298db61e07-kube-api-access-mhcmf" (OuterVolumeSpecName: "kube-api-access-mhcmf") pod "a73a0efe-90c0-4dfe-a76f-b7298db61e07" (UID: "a73a0efe-90c0-4dfe-a76f-b7298db61e07"). InnerVolumeSpecName "kube-api-access-mhcmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:54.389073 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.389032    2574 generic.go:356] "Generic (PLEG): container finished" podID="a73a0efe-90c0-4dfe-a76f-b7298db61e07" containerID="260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba" exitCode=0
Sep 16 20:00:54.389252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.389084    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"
Sep 16 20:00:54.389252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.389116    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4" event={"ID":"a73a0efe-90c0-4dfe-a76f-b7298db61e07","Type":"ContainerDied","Data":"260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba"}
Sep 16 20:00:54.389252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.389174    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4" event={"ID":"a73a0efe-90c0-4dfe-a76f-b7298db61e07","Type":"ContainerDied","Data":"1e6952822c15492a699b9edd42cd1a9b34580c7784b330429fbe35bfe4cb70c0"}
Sep 16 20:00:54.389252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.389205    2574 scope.go:117] "RemoveContainer" containerID="260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba"
Sep 16 20:00:54.401606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.401584    2574 scope.go:117] "RemoveContainer" containerID="260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba"
Sep 16 20:00:54.401911 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:00:54.401891    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba\": container with ID starting with 260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba not found: ID does not exist" containerID="260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba"
Sep 16 20:00:54.401981 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.401920    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba"} err="failed to get container status \"260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba\": rpc error: code = NotFound desc = could not find container \"260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba\": container with ID starting with 260b936a03c25ad0873a3f6e5bd62435d3127b23d9430a54bdb80a05c628cdba not found: ID does not exist"
Sep 16 20:00:54.434309 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.434278    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhcmf\" (UniqueName: \"kubernetes.io/projected/a73a0efe-90c0-4dfe-a76f-b7298db61e07-kube-api-access-mhcmf\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:00:54.434503 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.434311    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73a0efe-90c0-4dfe-a76f-b7298db61e07-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:00:54.434503 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.434338    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73a0efe-90c0-4dfe-a76f-b7298db61e07-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:00:54.438324 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.438294    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"]
Sep 16 20:00:54.441639 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:54.441614    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4"]
Sep 16 20:00:55.088239 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:55.088208    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a73a0efe-90c0-4dfe-a76f-b7298db61e07" path="/var/lib/kubelet/pods/a73a0efe-90c0-4dfe-a76f-b7298db61e07/volumes"
Sep 16 20:00:55.201971 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:00:55.201931    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-clxh4" podUID="a73a0efe-90c0-4dfe-a76f-b7298db61e07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 20:01:38.039895 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:38.039848    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:38.103738 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:38.103703    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:42.114088 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:42.114053    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:42.213804 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:42.213757    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:42.316581 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:42.316533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:46.994287 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:46.994257    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"]
Sep 16 20:01:46.994817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:46.994664    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73a0efe-90c0-4dfe-a76f-b7298db61e07" containerName="main"
Sep 16 20:01:46.994817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:46.994677    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73a0efe-90c0-4dfe-a76f-b7298db61e07" containerName="main"
Sep 16 20:01:46.994817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:46.994745    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73a0efe-90c0-4dfe-a76f-b7298db61e07" containerName="main"
Sep 16 20:01:46.998920 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:46.998904    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.001685 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.001667    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-c9sjl\""
Sep 16 20:01:47.001801 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.001739    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 20:01:47.002700 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.002681    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 20:01:47.002753 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.002701    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 20:01:47.010840 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.010819    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"]
Sep 16 20:01:47.013834 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.013810    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:47.084655 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.084622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31616609-36b4-4d10-a9f0-fefe67de7cca-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.084821 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.084731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.084821 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.084759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.084901 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.084830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.084901 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.084868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.084988 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.084967    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv9d2\" (UniqueName: \"kubernetes.io/projected/31616609-36b4-4d10-a9f0-fefe67de7cca-kube-api-access-zv9d2\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.186446 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.186388    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.186642 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.186453    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.186642 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.186499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.186642 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.186523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.186642 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.186548    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9d2\" (UniqueName: \"kubernetes.io/projected/31616609-36b4-4d10-a9f0-fefe67de7cca-kube-api-access-zv9d2\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.186642 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.186580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31616609-36b4-4d10-a9f0-fefe67de7cca-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.187011 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.186981    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.187100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.187014    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.187100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.187047    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.188756 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.188736    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.189137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.189117    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31616609-36b4-4d10-a9f0-fefe67de7cca-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.208255 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.208224    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9d2\" (UniqueName: \"kubernetes.io/projected/31616609-36b4-4d10-a9f0-fefe67de7cca-kube-api-access-zv9d2\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.309968 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.309878    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:47.395304 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.395180    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:47.589519 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.589480    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"]
Sep 16 20:01:47.592317 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:01:47.592289    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31616609_36b4_4d10_a9f0_fefe67de7cca.slice/crio-67c556aa43b209c7d8d7b90f2aaf3f57bc9b40b6c2011eb8cec655bd2c59e52b WatchSource:0}: Error finding container 67c556aa43b209c7d8d7b90f2aaf3f57bc9b40b6c2011eb8cec655bd2c59e52b: Status 404 returned error can't find the container with id 67c556aa43b209c7d8d7b90f2aaf3f57bc9b40b6c2011eb8cec655bd2c59e52b
Sep 16 20:01:47.678173 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.678137    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw" event={"ID":"31616609-36b4-4d10-a9f0-fefe67de7cca","Type":"ContainerStarted","Data":"a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988"}
Sep 16 20:01:47.678346 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.678200    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw" event={"ID":"31616609-36b4-4d10-a9f0-fefe67de7cca","Type":"ContainerStarted","Data":"67c556aa43b209c7d8d7b90f2aaf3f57bc9b40b6c2011eb8cec655bd2c59e52b"}
Sep 16 20:01:47.696343 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:47.696294    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw" podStartSLOduration=1.696275887 podStartE2EDuration="1.696275887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:47.694188461 +0000 UTC m=+1339.251533572" watchObservedRunningTime="2026-09-16 20:01:47.696275887 +0000 UTC m=+1339.253620973"
Sep 16 20:01:48.310934 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:48.310891    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:49.613218 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:49.613180    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:55.552891 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:55.552811    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:55.611059 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:55.611006    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:57.310530 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:57.310492    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:57.322926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:57.322894    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:57.739419 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:57.739380    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:01:58.608894 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:01:58.608848    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:05.197652 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:05.197601    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:20.941461 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:20.941422    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:21.088862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.088834    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"]
Sep 16 20:02:21.097548 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:02:21.097511    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:02:21.097548 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:02:21.097548 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:02:21.097548 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988" cmd=["/bin/sleep","15"]
Sep 16 20:02:21.097803 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.097557    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw" message="" err=<
Sep 16 20:02:21.097803 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:02:21.097803 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:02:21.097803 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:02:21.097803 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:02:21.097569    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:02:21.097803 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:02:21.097803 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:02:21.097803 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw" podUID="31616609-36b4-4d10-a9f0-fefe67de7cca" containerName="main" containerID="cri-o://a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988"
Sep 16 20:02:21.097803 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.097602    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw" podUID="31616609-36b4-4d10-a9f0-fefe67de7cca" containerName="main" containerID="cri-o://a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988" gracePeriod=60
Sep 16 20:02:21.100845 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.100819    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:21.193549 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.193447    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:21.729052 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.729013    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw" podUID="31616609-36b4-4d10-a9f0-fefe67de7cca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 20:02:21.847957 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.844550    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:02:21.859501 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.859469    2574 generic.go:356] "Generic (PLEG): container finished" podID="31616609-36b4-4d10-a9f0-fefe67de7cca" containerID="a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988" exitCode=0
Sep 16 20:02:21.859651 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.859512    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"
Sep 16 20:02:21.859651 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.859544    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw" event={"ID":"31616609-36b4-4d10-a9f0-fefe67de7cca","Type":"ContainerDied","Data":"a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988"}
Sep 16 20:02:21.859651 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.859610    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw" event={"ID":"31616609-36b4-4d10-a9f0-fefe67de7cca","Type":"ContainerDied","Data":"67c556aa43b209c7d8d7b90f2aaf3f57bc9b40b6c2011eb8cec655bd2c59e52b"}
Sep 16 20:02:21.859651 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.859633    2574 scope.go:117] "RemoveContainer" containerID="a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988"
Sep 16 20:02:21.869233 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.869209    2574 scope.go:117] "RemoveContainer" containerID="a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988"
Sep 16 20:02:21.869600 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:02:21.869558    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988\": container with ID starting with a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988 not found: ID does not exist" containerID="a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988"
Sep 16 20:02:21.869692 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.869612    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988"} err="failed to get container status \"a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988\": rpc error: code = NotFound desc = could not find container \"a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988\": container with ID starting with a30f06a26d52c54abd55dfac2a6bc28e30f07510030feb3f027ad1da28a86988 not found: ID does not exist"
Sep 16 20:02:21.885466 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.885432    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-model-cache\") pod \"31616609-36b4-4d10-a9f0-fefe67de7cca\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") "
Sep 16 20:02:21.885577 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.885488    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-dshm\") pod \"31616609-36b4-4d10-a9f0-fefe67de7cca\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") "
Sep 16 20:02:21.885619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.885606    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-tmp-dir\") pod \"31616609-36b4-4d10-a9f0-fefe67de7cca\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") "
Sep 16 20:02:21.885680 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.885639    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv9d2\" (UniqueName: \"kubernetes.io/projected/31616609-36b4-4d10-a9f0-fefe67de7cca-kube-api-access-zv9d2\") pod \"31616609-36b4-4d10-a9f0-fefe67de7cca\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") "
Sep 16 20:02:21.885680 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.885653    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-model-cache" (OuterVolumeSpecName: "model-cache") pod "31616609-36b4-4d10-a9f0-fefe67de7cca" (UID: "31616609-36b4-4d10-a9f0-fefe67de7cca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:21.885769 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.885699    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31616609-36b4-4d10-a9f0-fefe67de7cca-tls-certs\") pod \"31616609-36b4-4d10-a9f0-fefe67de7cca\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") "
Sep 16 20:02:21.885769 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.885725    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-home\") pod \"31616609-36b4-4d10-a9f0-fefe67de7cca\" (UID: \"31616609-36b4-4d10-a9f0-fefe67de7cca\") "
Sep 16 20:02:21.885911 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.885878    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31616609-36b4-4d10-a9f0-fefe67de7cca" (UID: "31616609-36b4-4d10-a9f0-fefe67de7cca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:21.886071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.886028    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-home" (OuterVolumeSpecName: "home") pod "31616609-36b4-4d10-a9f0-fefe67de7cca" (UID: "31616609-36b4-4d10-a9f0-fefe67de7cca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:21.886071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.886047    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:02:21.886071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.886067    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:02:21.887845 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.887818    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-dshm" (OuterVolumeSpecName: "dshm") pod "31616609-36b4-4d10-a9f0-fefe67de7cca" (UID: "31616609-36b4-4d10-a9f0-fefe67de7cca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:02:21.887944 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.887923    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31616609-36b4-4d10-a9f0-fefe67de7cca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31616609-36b4-4d10-a9f0-fefe67de7cca" (UID: "31616609-36b4-4d10-a9f0-fefe67de7cca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:02:21.887944 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.887926    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31616609-36b4-4d10-a9f0-fefe67de7cca-kube-api-access-zv9d2" (OuterVolumeSpecName: "kube-api-access-zv9d2") pod "31616609-36b4-4d10-a9f0-fefe67de7cca" (UID: "31616609-36b4-4d10-a9f0-fefe67de7cca"). InnerVolumeSpecName "kube-api-access-zv9d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:02:21.987321 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.987231    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:02:21.987321 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.987266    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv9d2\" (UniqueName: \"kubernetes.io/projected/31616609-36b4-4d10-a9f0-fefe67de7cca-kube-api-access-zv9d2\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:02:21.987321 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.987283    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31616609-36b4-4d10-a9f0-fefe67de7cca-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:02:21.987321 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:21.987297    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31616609-36b4-4d10-a9f0-fefe67de7cca-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:02:22.182689 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:22.182660    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"]
Sep 16 20:02:22.191750 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:22.191723    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-6lrpw"]
Sep 16 20:02:23.087915 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:23.087873    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31616609-36b4-4d10-a9f0-fefe67de7cca" path="/var/lib/kubelet/pods/31616609-36b4-4d10-a9f0-fefe67de7cca/volumes"
Sep 16 20:02:46.202176 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:46.202135    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:47.989206 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:02:47.989164    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:10.506608 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:10.506560    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:10.595561 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:10.595522    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:16.747114 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:16.747078    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:22.462943 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:22.462844    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:22.530322 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:22.530289    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:25.829069 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:25.829030    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:26.148225 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:26.148189    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:36.258159 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:36.258122    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:36.360541 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:36.360500    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:36.461533 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:36.461502    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:40.497213 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:40.497180    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:43.041688 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:43.041639    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:48.212921 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:48.212885    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:48.414293 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:48.414254    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:50.126949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:50.126914    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:50.201786 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:03:50.201734    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:03.422846 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:04:03.422790    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:07.107606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:04:07.107562    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:07.198587 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:04:07.198541    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:07.283301 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:04:07.283263    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:28.983317 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:04:28.983287    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:04:28.983898 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:04:28.983520    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:04:56.930839 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:04:56.930756    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:57.023977 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:04:57.023941    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:01.046179 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:01.046142    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:01.146462 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:01.146421    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:03.481840 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:03.481801    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:09.648536 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:09.648495    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:09.745810 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:09.745777    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:33.538489 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.538457    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"]
Sep 16 20:05:33.539052 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.538888    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31616609-36b4-4d10-a9f0-fefe67de7cca" containerName="main"
Sep 16 20:05:33.539052 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.538907    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="31616609-36b4-4d10-a9f0-fefe67de7cca" containerName="main"
Sep 16 20:05:33.539052 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.538989    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="31616609-36b4-4d10-a9f0-fefe67de7cca" containerName="main"
Sep 16 20:05:33.542694 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.542676    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.545230 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.545206    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 20:05:33.546240 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.546217    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-tgzfq\""
Sep 16 20:05:33.546346 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.546276    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 20:05:33.546346 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.546292    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-q82pp\""
Sep 16 20:05:33.546485 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.546370    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 20:05:33.555761 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.555740    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"]
Sep 16 20:05:33.583801 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.583764    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:33.609697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.609671    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-home\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.609801 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.609701    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.609801 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.609721    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714b5170-1011-4b7a-acd5-6dd993b53b46-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.609801 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.609737    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc99t\" (UniqueName: \"kubernetes.io/projected/714b5170-1011-4b7a-acd5-6dd993b53b46-kube-api-access-cc99t\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.609934 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.609799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.609934 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.609883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.710978 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.710945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.711141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.711006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-home\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.711141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.711028    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.711141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.711047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714b5170-1011-4b7a-acd5-6dd993b53b46-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.711141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.711076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc99t\" (UniqueName: \"kubernetes.io/projected/714b5170-1011-4b7a-acd5-6dd993b53b46-kube-api-access-cc99t\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.711141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.711101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.711457 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.711434    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.711518 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.711494    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.711568 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.711518    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-home\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.713341 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.713322    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.713654 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.713638    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714b5170-1011-4b7a-acd5-6dd993b53b46-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.722377 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.722354    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc99t\" (UniqueName: \"kubernetes.io/projected/714b5170-1011-4b7a-acd5-6dd993b53b46-kube-api-access-cc99t\") pod \"scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.852418 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.852374    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:33.859442 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:33.859395    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:34.129710 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:34.129682    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"]
Sep 16 20:05:34.130919 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:05:34.130894    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714b5170_1011_4b7a_acd5_6dd993b53b46.slice/crio-4080f65c32062715cecf1f43e12d9d8f0fe39aaa6c6c3eb430b85831d7c18a76 WatchSource:0}: Error finding container 4080f65c32062715cecf1f43e12d9d8f0fe39aaa6c6c3eb430b85831d7c18a76: Status 404 returned error can't find the container with id 4080f65c32062715cecf1f43e12d9d8f0fe39aaa6c6c3eb430b85831d7c18a76
Sep 16 20:05:34.133029 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:34.133011    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:05:34.836280 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:34.836235    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" event={"ID":"714b5170-1011-4b7a-acd5-6dd993b53b46","Type":"ContainerStarted","Data":"4080f65c32062715cecf1f43e12d9d8f0fe39aaa6c6c3eb430b85831d7c18a76"}
Sep 16 20:05:36.849653 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:36.849616    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" event={"ID":"714b5170-1011-4b7a-acd5-6dd993b53b46","Type":"ContainerStarted","Data":"6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d"}
Sep 16 20:05:36.850037 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:36.849759    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:37.856361 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:37.856329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" event={"ID":"714b5170-1011-4b7a-acd5-6dd993b53b46","Type":"ContainerStarted","Data":"095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954"}
Sep 16 20:05:37.879896 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:37.879847    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" podStartSLOduration=3.222944223 podStartE2EDuration="4.87983155s" totalImagesPullingTime="1.656887327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:05:37.87678534 +0000 UTC m=+1569.434130437" watchObservedRunningTime="2026-09-16 20:05:37.87983155 +0000 UTC m=+1569.437176635"
Sep 16 20:05:38.853116 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:38.853080    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:43.852967 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:43.852929    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:43.865297 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:43.865274    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:43.900077 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:43.900050    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:44.890477 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:44.890437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:54.905299 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:54.905269    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:05:59.291922 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:59.291887    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:59.389663 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:05:59.389630    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:06:13.018312 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.018276    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:06:13.054603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.054572    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"]
Sep 16 20:06:13.074284 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:06:13.074239    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:06:13.074284 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:06:13.074284 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:06:13.074284 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954" cmd=["/bin/sleep","15"]
Sep 16 20:06:13.074729 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.074304    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" message="" err=<
Sep 16 20:06:13.074729 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:06:13.074729 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:06:13.074729 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:06:13.074729 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:06:13.074319    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:06:13.074729 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:06:13.074729 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:06:13.074729 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerName="main" containerID="cri-o://095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954"
Sep 16 20:06:13.074729 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.074375    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerName="main" containerID="cri-o://095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954" gracePeriod=60
Sep 16 20:06:13.126295 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.126255    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:06:13.291311 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.291275    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerName="llm-d-routing-sidecar" containerID="cri-o://6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d" gracePeriod=60
Sep 16 20:06:13.543082 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.542997    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:06:13.669337 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669298    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714b5170-1011-4b7a-acd5-6dd993b53b46-tls-certs\") pod \"714b5170-1011-4b7a-acd5-6dd993b53b46\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") "
Sep 16 20:06:13.669337 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669343    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-dshm\") pod \"714b5170-1011-4b7a-acd5-6dd993b53b46\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") "
Sep 16 20:06:13.669612 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669385    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-tmp-dir\") pod \"714b5170-1011-4b7a-acd5-6dd993b53b46\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") "
Sep 16 20:06:13.669612 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669478    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-model-cache\") pod \"714b5170-1011-4b7a-acd5-6dd993b53b46\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") "
Sep 16 20:06:13.669612 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669580    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc99t\" (UniqueName: \"kubernetes.io/projected/714b5170-1011-4b7a-acd5-6dd993b53b46-kube-api-access-cc99t\") pod \"714b5170-1011-4b7a-acd5-6dd993b53b46\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") "
Sep 16 20:06:13.669775 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669653    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-home\") pod \"714b5170-1011-4b7a-acd5-6dd993b53b46\" (UID: \"714b5170-1011-4b7a-acd5-6dd993b53b46\") "
Sep 16 20:06:13.669836 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669794    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "714b5170-1011-4b7a-acd5-6dd993b53b46" (UID: "714b5170-1011-4b7a-acd5-6dd993b53b46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:13.669836 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669810    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-model-cache" (OuterVolumeSpecName: "model-cache") pod "714b5170-1011-4b7a-acd5-6dd993b53b46" (UID: "714b5170-1011-4b7a-acd5-6dd993b53b46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:13.669957 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669920    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-home" (OuterVolumeSpecName: "home") pod "714b5170-1011-4b7a-acd5-6dd993b53b46" (UID: "714b5170-1011-4b7a-acd5-6dd993b53b46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:13.669957 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669940    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.670063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.669956    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.671820 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.671785    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714b5170-1011-4b7a-acd5-6dd993b53b46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "714b5170-1011-4b7a-acd5-6dd993b53b46" (UID: "714b5170-1011-4b7a-acd5-6dd993b53b46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:06:13.671939 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.671832    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-dshm" (OuterVolumeSpecName: "dshm") pod "714b5170-1011-4b7a-acd5-6dd993b53b46" (UID: "714b5170-1011-4b7a-acd5-6dd993b53b46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:06:13.671939 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.671866    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714b5170-1011-4b7a-acd5-6dd993b53b46-kube-api-access-cc99t" (OuterVolumeSpecName: "kube-api-access-cc99t") pod "714b5170-1011-4b7a-acd5-6dd993b53b46" (UID: "714b5170-1011-4b7a-acd5-6dd993b53b46"). InnerVolumeSpecName "kube-api-access-cc99t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:06:13.771338 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.771304    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc99t\" (UniqueName: \"kubernetes.io/projected/714b5170-1011-4b7a-acd5-6dd993b53b46-kube-api-access-cc99t\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.771338 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.771339    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.771564 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.771353    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/714b5170-1011-4b7a-acd5-6dd993b53b46-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:06:13.771564 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:13.771365    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/714b5170-1011-4b7a-acd5-6dd993b53b46-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:06:14.045833 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.045798    2574 generic.go:356] "Generic (PLEG): container finished" podID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerID="095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954" exitCode=0
Sep 16 20:06:14.045833 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.045827    2574 generic.go:356] "Generic (PLEG): container finished" podID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerID="6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d" exitCode=0
Sep 16 20:06:14.046295 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.045845    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"
Sep 16 20:06:14.046295 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.045857    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" event={"ID":"714b5170-1011-4b7a-acd5-6dd993b53b46","Type":"ContainerDied","Data":"095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954"}
Sep 16 20:06:14.046295 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.045907    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" event={"ID":"714b5170-1011-4b7a-acd5-6dd993b53b46","Type":"ContainerDied","Data":"6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d"}
Sep 16 20:06:14.046295 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.045929    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm" event={"ID":"714b5170-1011-4b7a-acd5-6dd993b53b46","Type":"ContainerDied","Data":"4080f65c32062715cecf1f43e12d9d8f0fe39aaa6c6c3eb430b85831d7c18a76"}
Sep 16 20:06:14.046295 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.045948    2574 scope.go:117] "RemoveContainer" containerID="095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954"
Sep 16 20:06:14.056242 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.056222    2574 scope.go:117] "RemoveContainer" containerID="6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d"
Sep 16 20:06:14.064697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.064672    2574 scope.go:117] "RemoveContainer" containerID="095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954"
Sep 16 20:06:14.065025 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:06:14.064999    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954\": container with ID starting with 095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954 not found: ID does not exist" containerID="095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954"
Sep 16 20:06:14.065123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.065039    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954"} err="failed to get container status \"095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954\": rpc error: code = NotFound desc = could not find container \"095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954\": container with ID starting with 095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954 not found: ID does not exist"
Sep 16 20:06:14.065123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.065065    2574 scope.go:117] "RemoveContainer" containerID="6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d"
Sep 16 20:06:14.065389 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:06:14.065373    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d\": container with ID starting with 6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d not found: ID does not exist" containerID="6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d"
Sep 16 20:06:14.065461 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.065395    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d"} err="failed to get container status \"6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d\": rpc error: code = NotFound desc = could not find container \"6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d\": container with ID starting with 6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d not found: ID does not exist"
Sep 16 20:06:14.065461 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.065427    2574 scope.go:117] "RemoveContainer" containerID="095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954"
Sep 16 20:06:14.065711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.065686    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954"} err="failed to get container status \"095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954\": rpc error: code = NotFound desc = could not find container \"095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954\": container with ID starting with 095cef077bfcfff0679f97b30a746c022adf5c8669cece96299805db7b737954 not found: ID does not exist"
Sep 16 20:06:14.065711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.065710    2574 scope.go:117] "RemoveContainer" containerID="6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d"
Sep 16 20:06:14.066224 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.066167    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d"} err="failed to get container status \"6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d\": rpc error: code = NotFound desc = could not find container \"6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d\": container with ID starting with 6863f68f534f71bb69d4d0dc6df3822166fb81b1749783e0ef6fc3e7dab49a4d not found: ID does not exist"
Sep 16 20:06:14.068911 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.068889    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"]
Sep 16 20:06:14.074598 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:14.074577    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-7688675d4f-rxvbm"]
Sep 16 20:06:15.087506 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:15.087478    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" path="/var/lib/kubelet/pods/714b5170-1011-4b7a-acd5-6dd993b53b46/volumes"
Sep 16 20:06:46.603524 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:46.603486    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:06:46.708818 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:46.708777    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:06:46.809246 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:46.809203    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:06:47.700492 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:47.700443    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:06:47.808135 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:47.808103    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:06:50.004761 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:50.004716    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:06:54.013232 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:06:54.013194    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:00.061841 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:00.061803    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:00.123464 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:00.123433    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:03.235033 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:03.234996    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:03.354691 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:03.354646    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:09.717059 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:09.717027    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:15.863218 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:15.863187    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:15.956424 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:15.956366    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:17.711229 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:17.711199    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:47.486175 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.486108    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"]
Sep 16 20:07:47.486787 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.486479    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerName="main"
Sep 16 20:07:47.486787 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.486491    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerName="main"
Sep 16 20:07:47.486787 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.486502    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerName="llm-d-routing-sidecar"
Sep 16 20:07:47.486787 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.486508    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerName="llm-d-routing-sidecar"
Sep 16 20:07:47.486787 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.486562    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerName="llm-d-routing-sidecar"
Sep 16 20:07:47.486787 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.486575    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="714b5170-1011-4b7a-acd5-6dd993b53b46" containerName="main"
Sep 16 20:07:47.490131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.490113    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.493679 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.493651    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 20:07:47.493679 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.493666    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-8bf5h\""
Sep 16 20:07:47.493869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.493672    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-4j5x2\""
Sep 16 20:07:47.493869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.493726    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 20:07:47.493869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.493657    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 20:07:47.505701 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.505680    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"]
Sep 16 20:07:47.537288 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.537253    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:47.582443 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.582388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.582602 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.582472    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqlrg\" (UniqueName: \"kubernetes.io/projected/89de552f-c6f9-4389-987c-d2387c4a9b1b-kube-api-access-mqlrg\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.582602 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.582498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.582602 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.582523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.582726 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.582635    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.582726 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.582680    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89de552f-c6f9-4389-987c-d2387c4a9b1b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.684063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.684030    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.684227 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.684080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqlrg\" (UniqueName: \"kubernetes.io/projected/89de552f-c6f9-4389-987c-d2387c4a9b1b-kube-api-access-mqlrg\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.684227 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.684122    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.684227 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.684150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.684227 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.684208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.684467 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.684254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89de552f-c6f9-4389-987c-d2387c4a9b1b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.684467 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.684384    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.684577 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.684498    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.684650 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.684630    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.686483 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.686463    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.686667 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.686657    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89de552f-c6f9-4389-987c-d2387c4a9b1b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.694684 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.694662    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqlrg\" (UniqueName: \"kubernetes.io/projected/89de552f-c6f9-4389-987c-d2387c4a9b1b-kube-api-access-mqlrg\") pod \"scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.800064 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.799973    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:47.812261 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:47.812230    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:07:48.088514 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:48.088490    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"]
Sep 16 20:07:48.089954 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:07:48.089931    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89de552f_c6f9_4389_987c_d2387c4a9b1b.slice/crio-3c7f5dcac032caa83c02c93ff917bf374d24ece40feaf2cc85b10ef62cfb60a8 WatchSource:0}: Error finding container 3c7f5dcac032caa83c02c93ff917bf374d24ece40feaf2cc85b10ef62cfb60a8: Status 404 returned error can't find the container with id 3c7f5dcac032caa83c02c93ff917bf374d24ece40feaf2cc85b10ef62cfb60a8
Sep 16 20:07:48.529199 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:48.529090    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" event={"ID":"89de552f-c6f9-4389-987c-d2387c4a9b1b","Type":"ContainerStarted","Data":"93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51"}
Sep 16 20:07:48.529199 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:48.529161    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" event={"ID":"89de552f-c6f9-4389-987c-d2387c4a9b1b","Type":"ContainerStarted","Data":"3c7f5dcac032caa83c02c93ff917bf374d24ece40feaf2cc85b10ef62cfb60a8"}
Sep 16 20:07:48.529660 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:48.529284    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:49.535777 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:49.535696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" event={"ID":"89de552f-c6f9-4389-987c-d2387c4a9b1b","Type":"ContainerStarted","Data":"ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c"}
Sep 16 20:07:49.558990 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:49.558943    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" podStartSLOduration=2.558926269 podStartE2EDuration="2.558926269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:07:49.557948477 +0000 UTC m=+1701.115293597" watchObservedRunningTime="2026-09-16 20:07:49.558926269 +0000 UTC m=+1701.116271357"
Sep 16 20:07:49.800979 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:49.800905    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:57.800416 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:57.800314    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:57.812584 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:57.812560    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:58.592614 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:58.592581    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:58.593771 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:58.593750    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:07:59.802785 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:07:59.802755    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:08:28.621211 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:28.621161    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:08:28.641446 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:28.641416    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"]
Sep 16 20:08:28.651380 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:08:28.651346    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:08:28.651380 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:08:28.651380 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:08:28.651380 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c" cmd=["/bin/sleep","15"]
Sep 16 20:08:28.651654 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:28.651397    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" message="" err=<
Sep 16 20:08:28.651654 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:08:28.651654 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:08:28.651654 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:08:28.651654 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:08:28.651426    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:08:28.651654 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:08:28.651654 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:08:28.651654 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerName="main" containerID="cri-o://ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c"
Sep 16 20:08:28.651654 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:28.651468    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerName="main" containerID="cri-o://ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c" gracePeriod=60
Sep 16 20:08:28.693641 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:28.693599    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:08:28.868136 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:28.868104    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerName="llm-d-routing-sidecar" containerID="cri-o://93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51" gracePeriod=60
Sep 16 20:08:29.120422 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.120377    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:08:29.143663 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.143559    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqlrg\" (UniqueName: \"kubernetes.io/projected/89de552f-c6f9-4389-987c-d2387c4a9b1b-kube-api-access-mqlrg\") pod \"89de552f-c6f9-4389-987c-d2387c4a9b1b\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") "
Sep 16 20:08:29.143663 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.143652    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89de552f-c6f9-4389-987c-d2387c4a9b1b-tls-certs\") pod \"89de552f-c6f9-4389-987c-d2387c4a9b1b\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") "
Sep 16 20:08:29.143850 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.143747    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-dshm\") pod \"89de552f-c6f9-4389-987c-d2387c4a9b1b\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") "
Sep 16 20:08:29.143850 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.143787    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-tmp-dir\") pod \"89de552f-c6f9-4389-987c-d2387c4a9b1b\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") "
Sep 16 20:08:29.143953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.143853    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-model-cache\") pod \"89de552f-c6f9-4389-987c-d2387c4a9b1b\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") "
Sep 16 20:08:29.143953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.143898    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-home\") pod \"89de552f-c6f9-4389-987c-d2387c4a9b1b\" (UID: \"89de552f-c6f9-4389-987c-d2387c4a9b1b\") "
Sep 16 20:08:29.144507 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.144480    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-home" (OuterVolumeSpecName: "home") pod "89de552f-c6f9-4389-987c-d2387c4a9b1b" (UID: "89de552f-c6f9-4389-987c-d2387c4a9b1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:08:29.145485 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.145450    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89de552f-c6f9-4389-987c-d2387c4a9b1b" (UID: "89de552f-c6f9-4389-987c-d2387c4a9b1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:08:29.149274 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.149242    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89de552f-c6f9-4389-987c-d2387c4a9b1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89de552f-c6f9-4389-987c-d2387c4a9b1b" (UID: "89de552f-c6f9-4389-987c-d2387c4a9b1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:08:29.149480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.149457    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89de552f-c6f9-4389-987c-d2387c4a9b1b-kube-api-access-mqlrg" (OuterVolumeSpecName: "kube-api-access-mqlrg") pod "89de552f-c6f9-4389-987c-d2387c4a9b1b" (UID: "89de552f-c6f9-4389-987c-d2387c4a9b1b"). InnerVolumeSpecName "kube-api-access-mqlrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:08:29.156329 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.156289    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "89de552f-c6f9-4389-987c-d2387c4a9b1b" (UID: "89de552f-c6f9-4389-987c-d2387c4a9b1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:08:29.156963 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.156865    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-dshm" (OuterVolumeSpecName: "dshm") pod "89de552f-c6f9-4389-987c-d2387c4a9b1b" (UID: "89de552f-c6f9-4389-987c-d2387c4a9b1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:08:29.244768 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.244731    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89de552f-c6f9-4389-987c-d2387c4a9b1b-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.244768 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.244760    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.244768 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.244769    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.244768 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.244777    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.245021 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.244785    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89de552f-c6f9-4389-987c-d2387c4a9b1b-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.245021 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.244794    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqlrg\" (UniqueName: \"kubernetes.io/projected/89de552f-c6f9-4389-987c-d2387c4a9b1b-kube-api-access-mqlrg\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:08:29.754538 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.754497    2574 generic.go:356] "Generic (PLEG): container finished" podID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerID="ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c" exitCode=0
Sep 16 20:08:29.754538 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.754536    2574 generic.go:356] "Generic (PLEG): container finished" podID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerID="93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51" exitCode=0
Sep 16 20:08:29.755060 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.754570    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"
Sep 16 20:08:29.755060 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.754576    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" event={"ID":"89de552f-c6f9-4389-987c-d2387c4a9b1b","Type":"ContainerDied","Data":"ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c"}
Sep 16 20:08:29.755060 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.754684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" event={"ID":"89de552f-c6f9-4389-987c-d2387c4a9b1b","Type":"ContainerDied","Data":"93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51"}
Sep 16 20:08:29.755060 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.754711    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf" event={"ID":"89de552f-c6f9-4389-987c-d2387c4a9b1b","Type":"ContainerDied","Data":"3c7f5dcac032caa83c02c93ff917bf374d24ece40feaf2cc85b10ef62cfb60a8"}
Sep 16 20:08:29.755060 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.754743    2574 scope.go:117] "RemoveContainer" containerID="ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c"
Sep 16 20:08:29.764128 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.764112    2574 scope.go:117] "RemoveContainer" containerID="93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51"
Sep 16 20:08:29.771522 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.771498    2574 scope.go:117] "RemoveContainer" containerID="ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c"
Sep 16 20:08:29.771806 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:08:29.771782    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c\": container with ID starting with ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c not found: ID does not exist" containerID="ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c"
Sep 16 20:08:29.771881 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.771820    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c"} err="failed to get container status \"ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c\": rpc error: code = NotFound desc = could not find container \"ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c\": container with ID starting with ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c not found: ID does not exist"
Sep 16 20:08:29.771881 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.771844    2574 scope.go:117] "RemoveContainer" containerID="93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51"
Sep 16 20:08:29.772122 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:08:29.772095    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51\": container with ID starting with 93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51 not found: ID does not exist" containerID="93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51"
Sep 16 20:08:29.772207 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.772127    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51"} err="failed to get container status \"93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51\": rpc error: code = NotFound desc = could not find container \"93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51\": container with ID starting with 93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51 not found: ID does not exist"
Sep 16 20:08:29.772207 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.772143    2574 scope.go:117] "RemoveContainer" containerID="ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c"
Sep 16 20:08:29.772348 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.772332    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c"} err="failed to get container status \"ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c\": rpc error: code = NotFound desc = could not find container \"ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c\": container with ID starting with ab1225ca39bf90b33f79d7f57c20d69d1d59285ea5edad33247f95fa6161597c not found: ID does not exist"
Sep 16 20:08:29.772348 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.772348    2574 scope.go:117] "RemoveContainer" containerID="93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51"
Sep 16 20:08:29.772624 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.772600    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51"} err="failed to get container status \"93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51\": rpc error: code = NotFound desc = could not find container \"93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51\": container with ID starting with 93b866097d9b0de9a7d241a14d3f43a846c7ca0a48c2921f5904da772b9c0b51 not found: ID does not exist"
Sep 16 20:08:29.777643 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.777618    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"]
Sep 16 20:08:29.784609 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:29.784587    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-57b96c584d-n7lpf"]
Sep 16 20:08:31.087826 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:31.087794    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" path="/var/lib/kubelet/pods/89de552f-c6f9-4389-987c-d2387c4a9b1b/volumes"
Sep 16 20:08:39.842958 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:39.842926    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:08:44.320281 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:44.320247    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:08:50.428822 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:08:50.428785    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:17.568504 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:17.568466    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:17.667562 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:17.667517    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:20.816590 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:20.816554    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:20.960502 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:20.960437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:21.059215 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:21.059184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:24.052212 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:24.052174    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:25.899248 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:25.899209    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:29.017839 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:29.017806    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:09:29.018247 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:29.017917    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:09:29.759139 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:29.759095    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:29.815475 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:29.815438    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:38.721703 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.721670    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"]
Sep 16 20:09:38.722100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.722024    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerName="llm-d-routing-sidecar"
Sep 16 20:09:38.722100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.722035    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerName="llm-d-routing-sidecar"
Sep 16 20:09:38.722100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.722077    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerName="main"
Sep 16 20:09:38.722100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.722087    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerName="main"
Sep 16 20:09:38.722260 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.722173    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerName="llm-d-routing-sidecar"
Sep 16 20:09:38.722260 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.722190    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89de552f-c6f9-4389-987c-d2387c4a9b1b" containerName="main"
Sep 16 20:09:38.725681 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.725659    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.728351 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.728325    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:09:38.728481 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.728367    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-r7qzz\""
Sep 16 20:09:38.728481 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.728332    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 20:09:38.728785 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.728768    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 20:09:38.729329 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.729310    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 20:09:38.729456 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.729357    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 20:09:38.737305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.737284    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"]
Sep 16 20:09:38.738659 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.738630    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:38.835963 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.835931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrzxk\" (UniqueName: \"kubernetes.io/projected/00e9780e-8eb4-40f6-a751-b312bdb66319-kube-api-access-nrzxk\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.836091 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.835975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00e9780e-8eb4-40f6-a751-b312bdb66319-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.836091 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.836004    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-dshm\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.836091 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.836028    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-home\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.836091 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.836058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-model-cache\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.836231 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.836141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.836231 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.836200    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e9780e-8eb4-40f6-a751-b312bdb66319-tls-certs\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.836299 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.836243    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937299 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937493 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e9780e-8eb4-40f6-a751-b312bdb66319-tls-certs\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937493 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937360    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937493 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrzxk\" (UniqueName: \"kubernetes.io/projected/00e9780e-8eb4-40f6-a751-b312bdb66319-kube-api-access-nrzxk\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937493 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00e9780e-8eb4-40f6-a751-b312bdb66319-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937734 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-dshm\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937734 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-home\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937734 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-model-cache\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937734 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937696    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937955 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937791    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.937955 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937939    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-home\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.938044 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.937938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-model-cache\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.938215 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.938191    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00e9780e-8eb4-40f6-a751-b312bdb66319-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.939799 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.939776    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-dshm\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.940168 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.940148    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e9780e-8eb4-40f6-a751-b312bdb66319-tls-certs\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:38.946359 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:38.946340    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrzxk\" (UniqueName: \"kubernetes.io/projected/00e9780e-8eb4-40f6-a751-b312bdb66319-kube-api-access-nrzxk\") pod \"tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:39.015735 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:39.015648    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:39.036822 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:39.036794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:39.117567 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:39.117533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:39.312331 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:39.312308    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"]
Sep 16 20:09:39.314038 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:09:39.314017    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e9780e_8eb4_40f6_a751_b312bdb66319.slice/crio-7178f031e83aedbdc9890c93be2299bd08e625d93be906d1f0d7d1dd65a5aad2 WatchSource:0}: Error finding container 7178f031e83aedbdc9890c93be2299bd08e625d93be906d1f0d7d1dd65a5aad2: Status 404 returned error can't find the container with id 7178f031e83aedbdc9890c93be2299bd08e625d93be906d1f0d7d1dd65a5aad2
Sep 16 20:09:40.111132 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:40.111087    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" event={"ID":"00e9780e-8eb4-40f6-a751-b312bdb66319","Type":"ContainerStarted","Data":"bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf"}
Sep 16 20:09:40.111603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:40.111159    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" event={"ID":"00e9780e-8eb4-40f6-a751-b312bdb66319","Type":"ContainerStarted","Data":"7178f031e83aedbdc9890c93be2299bd08e625d93be906d1f0d7d1dd65a5aad2"}
Sep 16 20:09:42.123682 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:42.123647    2574 generic.go:356] "Generic (PLEG): container finished" podID="00e9780e-8eb4-40f6-a751-b312bdb66319" containerID="bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf" exitCode=0
Sep 16 20:09:42.124084 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:42.123723    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" event={"ID":"00e9780e-8eb4-40f6-a751-b312bdb66319","Type":"ContainerDied","Data":"bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf"}
Sep 16 20:09:43.129961 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:43.129923    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" event={"ID":"00e9780e-8eb4-40f6-a751-b312bdb66319","Type":"ContainerStarted","Data":"05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2"}
Sep 16 20:09:43.153517 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:43.153460    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" podStartSLOduration=2.899293625 podStartE2EDuration="5.153440453s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.254146828s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:09:43.152377582 +0000 UTC m=+1814.709722680" watchObservedRunningTime="2026-09-16 20:09:43.153440453 +0000 UTC m=+1814.710785541"
Sep 16 20:09:44.037889 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:44.037850    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:49.037372 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:49.037341    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:49.049845 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:49.049821    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:49.173023 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:49.172996    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:09:50.742497 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:50.742458    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:53.484905 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:53.484868    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:09:54.481022 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:09:54.480985    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:10:03.278036 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:03.278004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:10:11.795225 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.795186    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"]
Sep 16 20:10:11.798821 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.798799    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:11.801735 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.801713    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 20:10:11.803015 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.802984    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-pvnm5\""
Sep 16 20:10:11.803149 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.803011    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 20:10:11.803149 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.803116    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 20:10:11.813669 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.813643    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"]
Sep 16 20:10:11.816868 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.816844    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:10:11.926682 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.926648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88npr\" (UniqueName: \"kubernetes.io/projected/ab046e3b-a92f-46a1-bb4d-edad211cba2d-kube-api-access-88npr\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:11.926949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.926692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-dshm\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:11.926949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.926792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tls-certs\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:11.926949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.926839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-home\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:11.926949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.926892    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-model-cache\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:11.926949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:11.926924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.027848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.027812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tls-certs\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.028039 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.027853    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-home\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.028039 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.027877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-model-cache\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.028039 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.027896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.028039 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.028014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88npr\" (UniqueName: \"kubernetes.io/projected/ab046e3b-a92f-46a1-bb4d-edad211cba2d-kube-api-access-88npr\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.028271 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.028069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-dshm\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.028271 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.028258    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.028377 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.028316    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-home\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.028483 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.028459    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-model-cache\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.030250 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.030223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-dshm\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.030532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.030515    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tls-certs\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.037515 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.037478    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88npr\" (UniqueName: \"kubernetes.io/projected/ab046e3b-a92f-46a1-bb4d-edad211cba2d-kube-api-access-88npr\") pod \"test-div-v1-kserve-fb566d667-hzwgs\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.110841 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.110815    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:12.390443 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:12.390389    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"]
Sep 16 20:10:12.393331 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:10:12.393303    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab046e3b_a92f_46a1_bb4d_edad211cba2d.slice/crio-6368d602ec1b5a459b750f590b389d8797cf514a6df114e1d935d62658f93798 WatchSource:0}: Error finding container 6368d602ec1b5a459b750f590b389d8797cf514a6df114e1d935d62658f93798: Status 404 returned error can't find the container with id 6368d602ec1b5a459b750f590b389d8797cf514a6df114e1d935d62658f93798
Sep 16 20:10:13.292859 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:13.292820    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs" event={"ID":"ab046e3b-a92f-46a1-bb4d-edad211cba2d","Type":"ContainerStarted","Data":"6368d602ec1b5a459b750f590b389d8797cf514a6df114e1d935d62658f93798"}
Sep 16 20:10:14.299326 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:14.299292    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs" event={"ID":"ab046e3b-a92f-46a1-bb4d-edad211cba2d","Type":"ContainerStarted","Data":"4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b"}
Sep 16 20:10:14.321039 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:14.320990    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs" podStartSLOduration=2.102983203 podStartE2EDuration="3.320973365s" totalImagesPullingTime="1.217990162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:10:14.318597428 +0000 UTC m=+1845.875942530" watchObservedRunningTime="2026-09-16 20:10:14.320973365 +0000 UTC m=+1845.878318452"
Sep 16 20:10:15.111209 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:15.111164    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:21.788038 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.787993    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"]
Sep 16 20:10:21.793039 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.793014    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.799521 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.799493    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 20:10:21.800529 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.800500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-home\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.801318 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.801287    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55z7h\" (UniqueName: \"kubernetes.io/projected/5531aea9-aeef-4dae-9742-33d4e19b2479-kube-api-access-55z7h\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.801455 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.801352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.801522 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.801458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.801597 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.801552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.801658 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.801634    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5531aea9-aeef-4dae-9742-33d4e19b2479-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.803545 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.803524    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"]
Sep 16 20:10:21.825632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.825591    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:10:21.902699 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.902658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.902913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.902707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.902913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.902745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5531aea9-aeef-4dae-9742-33d4e19b2479-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.902913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.902778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-home\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.902913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.902829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55z7h\" (UniqueName: \"kubernetes.io/projected/5531aea9-aeef-4dae-9742-33d4e19b2479-kube-api-access-55z7h\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.902913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.902866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.903205 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.903117    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.903205 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.903179    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-home\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.903305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.903244    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.904959 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.904933    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.905317 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.905297    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5531aea9-aeef-4dae-9742-33d4e19b2479-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:21.911550 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:21.911525    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55z7h\" (UniqueName: \"kubernetes.io/projected/5531aea9-aeef-4dae-9742-33d4e19b2479-kube-api-access-55z7h\") pod \"test-div-v2-kserve-5f6fdb6b77-ms2vr\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:22.104905 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:22.104861    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:22.111908 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:22.111884    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:22.125510 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:22.125484    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:22.353720 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:22.353689    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:10:22.385694 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:22.385644    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"]
Sep 16 20:10:22.387696 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:10:22.387670    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5531aea9_aeef_4dae_9742_33d4e19b2479.slice/crio-99d98a956a74443c000e7bfe38505cb4c96cfdb466754f5bad90737a76e30822 WatchSource:0}: Error finding container 99d98a956a74443c000e7bfe38505cb4c96cfdb466754f5bad90737a76e30822: Status 404 returned error can't find the container with id 99d98a956a74443c000e7bfe38505cb4c96cfdb466754f5bad90737a76e30822
Sep 16 20:10:23.348730 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:23.348689    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr" event={"ID":"5531aea9-aeef-4dae-9742-33d4e19b2479","Type":"ContainerStarted","Data":"2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e"}
Sep 16 20:10:23.349190 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:23.348756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr" event={"ID":"5531aea9-aeef-4dae-9742-33d4e19b2479","Type":"ContainerStarted","Data":"99d98a956a74443c000e7bfe38505cb4c96cfdb466754f5bad90737a76e30822"}
Sep 16 20:10:23.368058 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:23.368011    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr" podStartSLOduration=2.367996294 podStartE2EDuration="2.367996294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:10:23.365879549 +0000 UTC m=+1854.923224682" watchObservedRunningTime="2026-09-16 20:10:23.367996294 +0000 UTC m=+1854.925341380"
Sep 16 20:10:24.105162 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:24.105124    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:25.201750 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:25.201714    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:10:32.105099 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:32.105063    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:32.121866 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:32.121816    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:10:32.415731 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:10:32.415643    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:11:05.492312 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:05.492228    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:05.510064 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:05.510033    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"]
Sep 16 20:11:05.520669 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:11:05.520628    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:11:05.520669 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:11:05.520669 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:11:05.520669 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2" cmd=["/bin/sleep","15"]
Sep 16 20:11:05.520913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:05.520692    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" message="" err=<
Sep 16 20:11:05.520913 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:11:05.520913 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:11:05.520913 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:11:05.520913 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:11:05.520710    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:11:05.520913 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:11:05.520913 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:11:05.520913 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" podUID="00e9780e-8eb4-40f6-a751-b312bdb66319" containerName="main" containerID="cri-o://05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2"
Sep 16 20:11:05.520913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:05.520753    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" podUID="00e9780e-8eb4-40f6-a751-b312bdb66319" containerName="main" containerID="cri-o://05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2" gracePeriod=60
Sep 16 20:11:05.578086 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:05.578050    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:05.893324 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:05.893298    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:11:06.014124 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014092    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrzxk\" (UniqueName: \"kubernetes.io/projected/00e9780e-8eb4-40f6-a751-b312bdb66319-kube-api-access-nrzxk\") pod \"00e9780e-8eb4-40f6-a751-b312bdb66319\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") "
Sep 16 20:11:06.014313 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014160    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-tmp-dir\") pod \"00e9780e-8eb4-40f6-a751-b312bdb66319\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") "
Sep 16 20:11:06.014313 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014187    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00e9780e-8eb4-40f6-a751-b312bdb66319-cabundle-cert\") pod \"00e9780e-8eb4-40f6-a751-b312bdb66319\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") "
Sep 16 20:11:06.014313 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014236    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-dshm\") pod \"00e9780e-8eb4-40f6-a751-b312bdb66319\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") "
Sep 16 20:11:06.014313 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014271    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e9780e-8eb4-40f6-a751-b312bdb66319-tls-certs\") pod \"00e9780e-8eb4-40f6-a751-b312bdb66319\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") "
Sep 16 20:11:06.014532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014455    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-kserve-provision-location\") pod \"00e9780e-8eb4-40f6-a751-b312bdb66319\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") "
Sep 16 20:11:06.014532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014485    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00e9780e-8eb4-40f6-a751-b312bdb66319" (UID: "00e9780e-8eb4-40f6-a751-b312bdb66319"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:06.014532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014526    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-home\") pod \"00e9780e-8eb4-40f6-a751-b312bdb66319\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") "
Sep 16 20:11:06.014755 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014557    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-model-cache\") pod \"00e9780e-8eb4-40f6-a751-b312bdb66319\" (UID: \"00e9780e-8eb4-40f6-a751-b312bdb66319\") "
Sep 16 20:11:06.014755 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014661    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e9780e-8eb4-40f6-a751-b312bdb66319-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00e9780e-8eb4-40f6-a751-b312bdb66319" (UID: "00e9780e-8eb4-40f6-a751-b312bdb66319"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:11:06.014974 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.014951    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-home" (OuterVolumeSpecName: "home") pod "00e9780e-8eb4-40f6-a751-b312bdb66319" (UID: "00e9780e-8eb4-40f6-a751-b312bdb66319"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:06.015071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.015019    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-model-cache" (OuterVolumeSpecName: "model-cache") pod "00e9780e-8eb4-40f6-a751-b312bdb66319" (UID: "00e9780e-8eb4-40f6-a751-b312bdb66319"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:06.015071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.015031    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.015071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.015053    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.015071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.015069    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00e9780e-8eb4-40f6-a751-b312bdb66319-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.016619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.016584    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-dshm" (OuterVolumeSpecName: "dshm") pod "00e9780e-8eb4-40f6-a751-b312bdb66319" (UID: "00e9780e-8eb4-40f6-a751-b312bdb66319"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:06.016619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.016614    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e9780e-8eb4-40f6-a751-b312bdb66319-kube-api-access-nrzxk" (OuterVolumeSpecName: "kube-api-access-nrzxk") pod "00e9780e-8eb4-40f6-a751-b312bdb66319" (UID: "00e9780e-8eb4-40f6-a751-b312bdb66319"). InnerVolumeSpecName "kube-api-access-nrzxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:11:06.016752 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.016649    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e9780e-8eb4-40f6-a751-b312bdb66319-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00e9780e-8eb4-40f6-a751-b312bdb66319" (UID: "00e9780e-8eb4-40f6-a751-b312bdb66319"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:11:06.042043 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.042013    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00e9780e-8eb4-40f6-a751-b312bdb66319" (UID: "00e9780e-8eb4-40f6-a751-b312bdb66319"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:06.116142 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.116109    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.116142 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.116136    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e9780e-8eb4-40f6-a751-b312bdb66319-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.116142 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.116149    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.116426 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.116157    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e9780e-8eb4-40f6-a751-b312bdb66319-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.116426 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.116166    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrzxk\" (UniqueName: \"kubernetes.io/projected/00e9780e-8eb4-40f6-a751-b312bdb66319-kube-api-access-nrzxk\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:06.585928 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.585892    2574 generic.go:356] "Generic (PLEG): container finished" podID="00e9780e-8eb4-40f6-a751-b312bdb66319" containerID="05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2" exitCode=0
Sep 16 20:11:06.586359 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.585947    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"
Sep 16 20:11:06.586359 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.586006    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" event={"ID":"00e9780e-8eb4-40f6-a751-b312bdb66319","Type":"ContainerDied","Data":"05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2"}
Sep 16 20:11:06.586359 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.586067    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v" event={"ID":"00e9780e-8eb4-40f6-a751-b312bdb66319","Type":"ContainerDied","Data":"7178f031e83aedbdc9890c93be2299bd08e625d93be906d1f0d7d1dd65a5aad2"}
Sep 16 20:11:06.586359 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.586104    2574 scope.go:117] "RemoveContainer" containerID="05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2"
Sep 16 20:11:06.596620 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.596601    2574 scope.go:117] "RemoveContainer" containerID="bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf"
Sep 16 20:11:06.605419 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.605380    2574 scope.go:117] "RemoveContainer" containerID="05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2"
Sep 16 20:11:06.605743 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:11:06.605723    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2\": container with ID starting with 05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2 not found: ID does not exist" containerID="05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2"
Sep 16 20:11:06.605794 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.605751    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2"} err="failed to get container status \"05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2\": rpc error: code = NotFound desc = could not find container \"05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2\": container with ID starting with 05ed98c7c2b300af430a258aaa8cee082280820df073aec0a751562c237936c2 not found: ID does not exist"
Sep 16 20:11:06.605794 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.605769    2574 scope.go:117] "RemoveContainer" containerID="bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf"
Sep 16 20:11:06.606026 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:11:06.606008    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf\": container with ID starting with bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf not found: ID does not exist" containerID="bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf"
Sep 16 20:11:06.606081 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.606036    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf"} err="failed to get container status \"bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf\": rpc error: code = NotFound desc = could not find container \"bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf\": container with ID starting with bb910656d14571b9d78ac3d44813e836b33368e8ed2fa271628b5cca27f09dcf not found: ID does not exist"
Sep 16 20:11:06.630187 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.630164    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"]
Sep 16 20:11:06.636965 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.636944    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-c4cf77cff-t5v4v"]
Sep 16 20:11:06.681504 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.681472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:06.781176 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.781137    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:06.883223 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:06.883129    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:07.051500 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:07.051462    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"]
Sep 16 20:11:07.089219 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:07.089188    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e9780e-8eb4-40f6-a751-b312bdb66319" path="/var/lib/kubelet/pods/00e9780e-8eb4-40f6-a751-b312bdb66319/volumes"
Sep 16 20:11:07.104629 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:07.104602    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:08.084918 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:08.084882    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"]
Sep 16 20:11:08.109138 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:08.109104    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:11.726279 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:11.726240    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:21.005611 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:21.005574    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:22.072245 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.072197    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr" podUID="5531aea9-aeef-4dae-9742-33d4e19b2479" containerName="main" containerID="cri-o://2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e" gracePeriod=45
Sep 16 20:11:22.332266 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.332245    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:11:22.466764 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.466729    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5531aea9-aeef-4dae-9742-33d4e19b2479-tls-certs\") pod \"5531aea9-aeef-4dae-9742-33d4e19b2479\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") "
Sep 16 20:11:22.466953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.466771    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-home\") pod \"5531aea9-aeef-4dae-9742-33d4e19b2479\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") "
Sep 16 20:11:22.466953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.466797    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-dshm\") pod \"5531aea9-aeef-4dae-9742-33d4e19b2479\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") "
Sep 16 20:11:22.466953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.466928    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-tmp-dir\") pod \"5531aea9-aeef-4dae-9742-33d4e19b2479\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") "
Sep 16 20:11:22.467129 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.466962    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55z7h\" (UniqueName: \"kubernetes.io/projected/5531aea9-aeef-4dae-9742-33d4e19b2479-kube-api-access-55z7h\") pod \"5531aea9-aeef-4dae-9742-33d4e19b2479\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") "
Sep 16 20:11:22.467129 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.466990    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-model-cache\") pod \"5531aea9-aeef-4dae-9742-33d4e19b2479\" (UID: \"5531aea9-aeef-4dae-9742-33d4e19b2479\") "
Sep 16 20:11:22.467129 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.467016    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-home" (OuterVolumeSpecName: "home") pod "5531aea9-aeef-4dae-9742-33d4e19b2479" (UID: "5531aea9-aeef-4dae-9742-33d4e19b2479"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:22.467265 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.467173    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5531aea9-aeef-4dae-9742-33d4e19b2479" (UID: "5531aea9-aeef-4dae-9742-33d4e19b2479"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:22.467312 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.467278    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-model-cache" (OuterVolumeSpecName: "model-cache") pod "5531aea9-aeef-4dae-9742-33d4e19b2479" (UID: "5531aea9-aeef-4dae-9742-33d4e19b2479"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:22.467312 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.467295    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:22.467452 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.467313    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:22.469111 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.469086    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-dshm" (OuterVolumeSpecName: "dshm") pod "5531aea9-aeef-4dae-9742-33d4e19b2479" (UID: "5531aea9-aeef-4dae-9742-33d4e19b2479"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:22.469249 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.469144    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5531aea9-aeef-4dae-9742-33d4e19b2479-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5531aea9-aeef-4dae-9742-33d4e19b2479" (UID: "5531aea9-aeef-4dae-9742-33d4e19b2479"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:11:22.469249 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.469161    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5531aea9-aeef-4dae-9742-33d4e19b2479-kube-api-access-55z7h" (OuterVolumeSpecName: "kube-api-access-55z7h") pod "5531aea9-aeef-4dae-9742-33d4e19b2479" (UID: "5531aea9-aeef-4dae-9742-33d4e19b2479"). InnerVolumeSpecName "kube-api-access-55z7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:11:22.568023 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.567985    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55z7h\" (UniqueName: \"kubernetes.io/projected/5531aea9-aeef-4dae-9742-33d4e19b2479-kube-api-access-55z7h\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:22.568023 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.568018    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:22.568023 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.568027    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5531aea9-aeef-4dae-9742-33d4e19b2479-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:22.568260 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.568035    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5531aea9-aeef-4dae-9742-33d4e19b2479-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:22.670305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.670226    2574 generic.go:356] "Generic (PLEG): container finished" podID="5531aea9-aeef-4dae-9742-33d4e19b2479" containerID="2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e" exitCode=0
Sep 16 20:11:22.670305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.670270    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr" event={"ID":"5531aea9-aeef-4dae-9742-33d4e19b2479","Type":"ContainerDied","Data":"2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e"}
Sep 16 20:11:22.670575 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.670328    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr" event={"ID":"5531aea9-aeef-4dae-9742-33d4e19b2479","Type":"ContainerDied","Data":"99d98a956a74443c000e7bfe38505cb4c96cfdb466754f5bad90737a76e30822"}
Sep 16 20:11:22.670575 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.670359    2574 scope.go:117] "RemoveContainer" containerID="2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e"
Sep 16 20:11:22.670575 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.670277    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"
Sep 16 20:11:22.680276 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.680260    2574 scope.go:117] "RemoveContainer" containerID="2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e"
Sep 16 20:11:22.680602 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:11:22.680581    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e\": container with ID starting with 2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e not found: ID does not exist" containerID="2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e"
Sep 16 20:11:22.680685 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.680614    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e"} err="failed to get container status \"2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e\": rpc error: code = NotFound desc = could not find container \"2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e\": container with ID starting with 2b2e55c17d6ed4cc9786c6e8b2d993d50a282731adf47e61d42565d511bb138e not found: ID does not exist"
Sep 16 20:11:22.692852 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.692829    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"]
Sep 16 20:11:22.696801 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:22.696775    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-ms2vr"]
Sep 16 20:11:23.089013 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.088981    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5531aea9-aeef-4dae-9742-33d4e19b2479" path="/var/lib/kubelet/pods/5531aea9-aeef-4dae-9742-33d4e19b2479/volumes"
Sep 16 20:11:23.103808 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.103774    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs" podUID="ab046e3b-a92f-46a1-bb4d-edad211cba2d" containerName="main" containerID="cri-o://4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b" gracePeriod=45
Sep 16 20:11:23.351751 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.351680    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:11:23.474644 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.474612    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-dshm\") pod \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") "
Sep 16 20:11:23.474644 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.474656    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-home\") pod \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") "
Sep 16 20:11:23.474904 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.474691    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88npr\" (UniqueName: \"kubernetes.io/projected/ab046e3b-a92f-46a1-bb4d-edad211cba2d-kube-api-access-88npr\") pod \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") "
Sep 16 20:11:23.474904 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.474710    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-model-cache\") pod \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") "
Sep 16 20:11:23.474904 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.474729    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tls-certs\") pod \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") "
Sep 16 20:11:23.474904 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.474765    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tmp-dir\") pod \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\" (UID: \"ab046e3b-a92f-46a1-bb4d-edad211cba2d\") "
Sep 16 20:11:23.475128 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.475088    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab046e3b-a92f-46a1-bb4d-edad211cba2d" (UID: "ab046e3b-a92f-46a1-bb4d-edad211cba2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:23.475128 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.475115    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-home" (OuterVolumeSpecName: "home") pod "ab046e3b-a92f-46a1-bb4d-edad211cba2d" (UID: "ab046e3b-a92f-46a1-bb4d-edad211cba2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:23.475429 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.475382    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab046e3b-a92f-46a1-bb4d-edad211cba2d" (UID: "ab046e3b-a92f-46a1-bb4d-edad211cba2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:23.476933 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.476906    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-dshm" (OuterVolumeSpecName: "dshm") pod "ab046e3b-a92f-46a1-bb4d-edad211cba2d" (UID: "ab046e3b-a92f-46a1-bb4d-edad211cba2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:11:23.477027 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.476924    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab046e3b-a92f-46a1-bb4d-edad211cba2d" (UID: "ab046e3b-a92f-46a1-bb4d-edad211cba2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:11:23.477027 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.476935    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab046e3b-a92f-46a1-bb4d-edad211cba2d-kube-api-access-88npr" (OuterVolumeSpecName: "kube-api-access-88npr") pod "ab046e3b-a92f-46a1-bb4d-edad211cba2d" (UID: "ab046e3b-a92f-46a1-bb4d-edad211cba2d"). InnerVolumeSpecName "kube-api-access-88npr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:11:23.575684 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.575644    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:23.575684 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.575677    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:23.575907 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.575714    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88npr\" (UniqueName: \"kubernetes.io/projected/ab046e3b-a92f-46a1-bb4d-edad211cba2d-kube-api-access-88npr\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:23.575907 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.575729    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:23.575907 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.575743    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:23.575907 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.575754    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab046e3b-a92f-46a1-bb4d-edad211cba2d-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:11:23.676931 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.676822    2574 generic.go:356] "Generic (PLEG): container finished" podID="ab046e3b-a92f-46a1-bb4d-edad211cba2d" containerID="4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b" exitCode=0
Sep 16 20:11:23.676931 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.676895    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"
Sep 16 20:11:23.676931 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.676915    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs" event={"ID":"ab046e3b-a92f-46a1-bb4d-edad211cba2d","Type":"ContainerDied","Data":"4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b"}
Sep 16 20:11:23.677217 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.677011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs" event={"ID":"ab046e3b-a92f-46a1-bb4d-edad211cba2d","Type":"ContainerDied","Data":"6368d602ec1b5a459b750f590b389d8797cf514a6df114e1d935d62658f93798"}
Sep 16 20:11:23.677217 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.677044    2574 scope.go:117] "RemoveContainer" containerID="4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b"
Sep 16 20:11:23.690204 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.690183    2574 scope.go:117] "RemoveContainer" containerID="4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b"
Sep 16 20:11:23.690498 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:11:23.690476    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b\": container with ID starting with 4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b not found: ID does not exist" containerID="4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b"
Sep 16 20:11:23.690569 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.690507    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b"} err="failed to get container status \"4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b\": rpc error: code = NotFound desc = could not find container \"4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b\": container with ID starting with 4febaf5d32d22dbab6ece975e31fe43c52cde30ba906003b85dc616e9a7aeb0b not found: ID does not exist"
Sep 16 20:11:23.699497 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.699472    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"]
Sep 16 20:11:23.703068 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:23.703044    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs"]
Sep 16 20:11:24.343610 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:24.343559    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-hzwgs" podUID="ab046e3b-a92f-46a1-bb4d-edad211cba2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 20:11:25.088679 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:25.088645    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab046e3b-a92f-46a1-bb4d-edad211cba2d" path="/var/lib/kubelet/pods/ab046e3b-a92f-46a1-bb4d-edad211cba2d/volumes"
Sep 16 20:11:29.182865 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.182830    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"]
Sep 16 20:11:29.183389 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183354    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab046e3b-a92f-46a1-bb4d-edad211cba2d" containerName="main"
Sep 16 20:11:29.183389 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183380    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab046e3b-a92f-46a1-bb4d-edad211cba2d" containerName="main"
Sep 16 20:11:29.183389 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183396    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e9780e-8eb4-40f6-a751-b312bdb66319" containerName="main"
Sep 16 20:11:29.183660 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183416    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e9780e-8eb4-40f6-a751-b312bdb66319" containerName="main"
Sep 16 20:11:29.183660 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183441    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5531aea9-aeef-4dae-9742-33d4e19b2479" containerName="main"
Sep 16 20:11:29.183660 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183446    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5531aea9-aeef-4dae-9742-33d4e19b2479" containerName="main"
Sep 16 20:11:29.183660 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183458    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e9780e-8eb4-40f6-a751-b312bdb66319" containerName="storage-initializer"
Sep 16 20:11:29.183660 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183466    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e9780e-8eb4-40f6-a751-b312bdb66319" containerName="storage-initializer"
Sep 16 20:11:29.183660 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183526    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab046e3b-a92f-46a1-bb4d-edad211cba2d" containerName="main"
Sep 16 20:11:29.183660 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183536    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e9780e-8eb4-40f6-a751-b312bdb66319" containerName="main"
Sep 16 20:11:29.183660 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.183547    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5531aea9-aeef-4dae-9742-33d4e19b2479" containerName="main"
Sep 16 20:11:29.187935 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.187918    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.191727 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.191708    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 20:11:29.191853 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.191710    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-zlvf9\""
Sep 16 20:11:29.191853 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.191710    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 20:11:29.191977 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.191710    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 20:11:29.199252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.199231    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"]
Sep 16 20:11:29.292224 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.292187    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:29.325198 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.325164    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7gn\" (UniqueName: \"kubernetes.io/projected/9c4a165b-e865-4d5a-990a-1514ca04c2d0-kube-api-access-vb7gn\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.325368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.325224    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.325368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.325252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-home\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.325368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.325295    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.325368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.325339    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.325368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.325356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.426363 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.426275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.426363 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.426313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-home\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.426363 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.426349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.426681 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.426370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.426681 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.426387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.426681 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.426445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7gn\" (UniqueName: \"kubernetes.io/projected/9c4a165b-e865-4d5a-990a-1514ca04c2d0-kube-api-access-vb7gn\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.426858 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.426832    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.427041 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.426841    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-home\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.427041 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.426908    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.428665 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.428641    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.428949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.428928    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.434619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.434589    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7gn\" (UniqueName: \"kubernetes.io/projected/9c4a165b-e865-4d5a-990a-1514ca04c2d0-kube-api-access-vb7gn\") pod \"test-div-v2-kserve-5f6fdb6b77-kg7zh\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.498482 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.498445    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:29.777348 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:29.777322    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"]
Sep 16 20:11:29.779425 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:11:29.779380    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4a165b_e865_4d5a_990a_1514ca04c2d0.slice/crio-6d537370735d75f5895e8d19ce7e014c20497cf987a26dd19eb68aabf6db79d7 WatchSource:0}: Error finding container 6d537370735d75f5895e8d19ce7e014c20497cf987a26dd19eb68aabf6db79d7: Status 404 returned error can't find the container with id 6d537370735d75f5895e8d19ce7e014c20497cf987a26dd19eb68aabf6db79d7
Sep 16 20:11:30.716433 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:30.716381    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh" event={"ID":"9c4a165b-e865-4d5a-990a-1514ca04c2d0","Type":"ContainerStarted","Data":"3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81"}
Sep 16 20:11:30.716814 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:30.716464    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh" event={"ID":"9c4a165b-e865-4d5a-990a-1514ca04c2d0","Type":"ContainerStarted","Data":"6d537370735d75f5895e8d19ce7e014c20497cf987a26dd19eb68aabf6db79d7"}
Sep 16 20:11:30.735128 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:30.735066    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh" podStartSLOduration=1.735048328 podStartE2EDuration="1.735048328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:11:30.733680814 +0000 UTC m=+1922.291025919" watchObservedRunningTime="2026-09-16 20:11:30.735048328 +0000 UTC m=+1922.292393456"
Sep 16 20:11:31.499321 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:31.499284    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:39.498926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:39.498882    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:39.511300 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:39.511275    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:39.775283 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:39.775195    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:11:49.042540 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:49.042499    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:51.287784 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:51.287745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:51.447469 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:51.447428    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:51.542978 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:51.542892    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:51.787651 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:51.787621    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"]
Sep 16 20:11:51.794046 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:51.793971    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:52.059532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:52.059428    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:52.204901 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:52.204859    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:52.328540 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:52.328446    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:55.005346 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:55.005307    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:55.106800 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:55.106760    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:55.387645 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:55.387607    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:55.499302 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:55.499262    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:56.366233 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:56.366197    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:58.652874 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:58.652837    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:11:59.908259 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:11:59.908219    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:02.386243 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:02.386212    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:04.567447 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:04.567410    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:04.666473 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:04.666436    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:06.843361 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:06.843309    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh" podUID="9c4a165b-e865-4d5a-990a-1514ca04c2d0" containerName="main" containerID="cri-o://3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81" gracePeriod=45
Sep 16 20:12:07.093551 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.093492    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:12:07.167653 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.167616    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-model-cache\") pod \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") "
Sep 16 20:12:07.167827 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.167660    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-dshm\") pod \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") "
Sep 16 20:12:07.167827 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.167700    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tmp-dir\") pod \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") "
Sep 16 20:12:07.167827 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.167767    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tls-certs\") pod \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") "
Sep 16 20:12:07.167827 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.167813    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb7gn\" (UniqueName: \"kubernetes.io/projected/9c4a165b-e865-4d5a-990a-1514ca04c2d0-kube-api-access-vb7gn\") pod \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") "
Sep 16 20:12:07.168061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.167839    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-home\") pod \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\" (UID: \"9c4a165b-e865-4d5a-990a-1514ca04c2d0\") "
Sep 16 20:12:07.168061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.167929    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c4a165b-e865-4d5a-990a-1514ca04c2d0" (UID: "9c4a165b-e865-4d5a-990a-1514ca04c2d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:07.168061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.167981    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c4a165b-e865-4d5a-990a-1514ca04c2d0" (UID: "9c4a165b-e865-4d5a-990a-1514ca04c2d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:07.168217 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.168206    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:12:07.168276 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.168227    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:12:07.168276 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.168196    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-home" (OuterVolumeSpecName: "home") pod "9c4a165b-e865-4d5a-990a-1514ca04c2d0" (UID: "9c4a165b-e865-4d5a-990a-1514ca04c2d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:07.170046 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.170024    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-dshm" (OuterVolumeSpecName: "dshm") pod "9c4a165b-e865-4d5a-990a-1514ca04c2d0" (UID: "9c4a165b-e865-4d5a-990a-1514ca04c2d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:12:07.170160 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.170114    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c4a165b-e865-4d5a-990a-1514ca04c2d0" (UID: "9c4a165b-e865-4d5a-990a-1514ca04c2d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:12:07.170224 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.170192    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4a165b-e865-4d5a-990a-1514ca04c2d0-kube-api-access-vb7gn" (OuterVolumeSpecName: "kube-api-access-vb7gn") pod "9c4a165b-e865-4d5a-990a-1514ca04c2d0" (UID: "9c4a165b-e865-4d5a-990a-1514ca04c2d0"). InnerVolumeSpecName "kube-api-access-vb7gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:12:07.269475 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.269439    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4a165b-e865-4d5a-990a-1514ca04c2d0-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:12:07.269475 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.269472    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb7gn\" (UniqueName: \"kubernetes.io/projected/9c4a165b-e865-4d5a-990a-1514ca04c2d0-kube-api-access-vb7gn\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:12:07.269475 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.269482    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:12:07.269715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.269491    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4a165b-e865-4d5a-990a-1514ca04c2d0-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:12:07.911611 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.911577    2574 generic.go:356] "Generic (PLEG): container finished" podID="9c4a165b-e865-4d5a-990a-1514ca04c2d0" containerID="3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81" exitCode=0
Sep 16 20:12:07.912022 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.911623    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"
Sep 16 20:12:07.912022 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.911632    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh" event={"ID":"9c4a165b-e865-4d5a-990a-1514ca04c2d0","Type":"ContainerDied","Data":"3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81"}
Sep 16 20:12:07.912022 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.911676    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh" event={"ID":"9c4a165b-e865-4d5a-990a-1514ca04c2d0","Type":"ContainerDied","Data":"6d537370735d75f5895e8d19ce7e014c20497cf987a26dd19eb68aabf6db79d7"}
Sep 16 20:12:07.912022 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.911697    2574 scope.go:117] "RemoveContainer" containerID="3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81"
Sep 16 20:12:07.920728 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.920708    2574 scope.go:117] "RemoveContainer" containerID="3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81"
Sep 16 20:12:07.920998 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:12:07.920980    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81\": container with ID starting with 3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81 not found: ID does not exist" containerID="3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81"
Sep 16 20:12:07.921040 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.921009    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81"} err="failed to get container status \"3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81\": rpc error: code = NotFound desc = could not find container \"3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81\": container with ID starting with 3e96be388dde41d39b728ca36c5e0997e9fe616d23e71d30c91e4dd0e9ebfc81 not found: ID does not exist"
Sep 16 20:12:07.938414 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.938382    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"]
Sep 16 20:12:07.944565 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:07.944543    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-kg7zh"]
Sep 16 20:12:08.238693 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:08.238606    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:08.338772 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:08.338717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:09.094281 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:09.094246    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c4a165b-e865-4d5a-990a-1514ca04c2d0" path="/var/lib/kubelet/pods/9c4a165b-e865-4d5a-990a-1514ca04c2d0/volumes"
Sep 16 20:12:12.521319 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:12.521280    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:18.499614 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:18.499578    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:18.585257 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:18.585225    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:22.558345 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:22.558246    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:34.707957 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.707922    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"]
Sep 16 20:12:34.708480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.708390    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4a165b-e865-4d5a-990a-1514ca04c2d0" containerName="main"
Sep 16 20:12:34.708480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.708420    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4a165b-e865-4d5a-990a-1514ca04c2d0" containerName="main"
Sep 16 20:12:34.708623 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.708505    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c4a165b-e865-4d5a-990a-1514ca04c2d0" containerName="main"
Sep 16 20:12:34.711441 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.711424    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.713993 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.713970    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 20:12:34.714118 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.713970    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:12:34.714873 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.714835    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 20:12:34.714981 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.714874    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 20:12:34.714981 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.714891    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-xzlb2\""
Sep 16 20:12:34.714981 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.714956    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 20:12:34.720657 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.720623    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:34.728502 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.728482    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"]
Sep 16 20:12:34.819833 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.819795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7f09fbf-5273-4a2b-a234-951b1d50f056-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.819833 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.819835    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.820084 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.819857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.820084 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.819953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f09fbf-5273-4a2b-a234-951b1d50f056-tls-certs\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.820084 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.819990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-home\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.820084 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.820055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-dshm\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.820251 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.820104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-model-cache\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.820251 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.820128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptgp\" (UniqueName: \"kubernetes.io/projected/f7f09fbf-5273-4a2b-a234-951b1d50f056-kube-api-access-wptgp\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.920765 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.920722    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7f09fbf-5273-4a2b-a234-951b1d50f056-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.920969 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.920772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.920969 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.920895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921103 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.920982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f09fbf-5273-4a2b-a234-951b1d50f056-tls-certs\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921103 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.921020    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-home\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921103 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.921056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-dshm\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921266 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.921109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-model-cache\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921266 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.921145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptgp\" (UniqueName: \"kubernetes.io/projected/f7f09fbf-5273-4a2b-a234-951b1d50f056-kube-api-access-wptgp\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921266 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.921198    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921266 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.921236    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921487 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.921467    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-home\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921537 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.921491    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-model-cache\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.921637 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.921615    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7f09fbf-5273-4a2b-a234-951b1d50f056-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.923319 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.923293    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-dshm\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.923610 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.923594    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f09fbf-5273-4a2b-a234-951b1d50f056-tls-certs\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:34.935277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:34.935251    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptgp\" (UniqueName: \"kubernetes.io/projected/f7f09fbf-5273-4a2b-a234-951b1d50f056-kube-api-access-wptgp\") pod \"tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:35.016902 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:35.016805    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:35.021543 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:35.021508    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:35.109610 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:35.109578    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:12:35.306026 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:35.305996    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"]
Sep 16 20:12:35.308224 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:12:35.308194    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f09fbf_5273_4a2b_a234_951b1d50f056.slice/crio-aea1412de5a6f64580ead8c90335ee7ce36a555fbbe6ae241cf02e3128902f3e WatchSource:0}: Error finding container aea1412de5a6f64580ead8c90335ee7ce36a555fbbe6ae241cf02e3128902f3e: Status 404 returned error can't find the container with id aea1412de5a6f64580ead8c90335ee7ce36a555fbbe6ae241cf02e3128902f3e
Sep 16 20:12:36.066784 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:36.066743    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" event={"ID":"f7f09fbf-5273-4a2b-a234-951b1d50f056","Type":"ContainerStarted","Data":"9f3ce2a038f46150dc4556b3efb7874207f67c9a8bd3a0c70be5b36b26ef9608"}
Sep 16 20:12:36.067261 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:36.066811    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" event={"ID":"f7f09fbf-5273-4a2b-a234-951b1d50f056","Type":"ContainerStarted","Data":"aea1412de5a6f64580ead8c90335ee7ce36a555fbbe6ae241cf02e3128902f3e"}
Sep 16 20:12:38.081800 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:38.081755    2574 generic.go:356] "Generic (PLEG): container finished" podID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerID="9f3ce2a038f46150dc4556b3efb7874207f67c9a8bd3a0c70be5b36b26ef9608" exitCode=0
Sep 16 20:12:38.082192 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:38.081827    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" event={"ID":"f7f09fbf-5273-4a2b-a234-951b1d50f056","Type":"ContainerDied","Data":"9f3ce2a038f46150dc4556b3efb7874207f67c9a8bd3a0c70be5b36b26ef9608"}
Sep 16 20:12:39.088830 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:39.088798    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" event={"ID":"f7f09fbf-5273-4a2b-a234-951b1d50f056","Type":"ContainerStarted","Data":"7a7803946189c44dbcc406063370acef81f923dfe194a1def6797de7af259cab"}
Sep 16 20:12:39.112160 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:39.112104    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" podStartSLOduration=2.77658946 podStartE2EDuration="5.112087557s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.335498097s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:12:39.111873196 +0000 UTC m=+1990.669218304" watchObservedRunningTime="2026-09-16 20:12:39.112087557 +0000 UTC m=+1990.669432644"
Sep 16 20:12:40.022625 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:40.022590    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:45.022256 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:45.022213    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:45.035174 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:45.035141    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:45.131964 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:45.131931    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:12:54.672096 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:12:54.672055    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:13:33.625167 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:33.625125    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:13:40.038122 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.038084    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:13:40.082464 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.082432    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"]
Sep 16 20:13:40.091450 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:13:40.091413    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:13:40.091450 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:13:40.091450 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:13:40.091450 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="7a7803946189c44dbcc406063370acef81f923dfe194a1def6797de7af259cab" cmd=["/bin/sleep","15"]
Sep 16 20:13:40.091706 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.091467    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" message="" err=<
Sep 16 20:13:40.091706 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:13:40.091706 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:13:40.091706 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:13:40.091706 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:13:40.091480    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:13:40.091706 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:13:40.091706 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:13:40.091706 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" podUID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerName="main" containerID="cri-o://7a7803946189c44dbcc406063370acef81f923dfe194a1def6797de7af259cab"
Sep 16 20:13:40.091706 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.091514    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" podUID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerName="main" containerID="cri-o://7a7803946189c44dbcc406063370acef81f923dfe194a1def6797de7af259cab" gracePeriod=60
Sep 16 20:13:40.115297 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.115262    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:13:40.121340 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.121311    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" podUID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 16 20:13:40.418105 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.418075    2574 generic.go:356] "Generic (PLEG): container finished" podID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerID="7a7803946189c44dbcc406063370acef81f923dfe194a1def6797de7af259cab" exitCode=0
Sep 16 20:13:40.418279 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.418151    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" event={"ID":"f7f09fbf-5273-4a2b-a234-951b1d50f056","Type":"ContainerDied","Data":"7a7803946189c44dbcc406063370acef81f923dfe194a1def6797de7af259cab"}
Sep 16 20:13:40.444766 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.444745    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:13:40.498811 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.498778    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-model-cache\") pod \"f7f09fbf-5273-4a2b-a234-951b1d50f056\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") "
Sep 16 20:13:40.498999 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.498824    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7f09fbf-5273-4a2b-a234-951b1d50f056-cabundle-cert\") pod \"f7f09fbf-5273-4a2b-a234-951b1d50f056\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") "
Sep 16 20:13:40.498999 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.498877    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-kserve-provision-location\") pod \"f7f09fbf-5273-4a2b-a234-951b1d50f056\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") "
Sep 16 20:13:40.498999 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.498927    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-tmp-dir\") pod \"f7f09fbf-5273-4a2b-a234-951b1d50f056\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") "
Sep 16 20:13:40.498999 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.498996    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-home\") pod \"f7f09fbf-5273-4a2b-a234-951b1d50f056\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") "
Sep 16 20:13:40.499225 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.499037    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f09fbf-5273-4a2b-a234-951b1d50f056-tls-certs\") pod \"f7f09fbf-5273-4a2b-a234-951b1d50f056\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") "
Sep 16 20:13:40.499225 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.499068    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7f09fbf-5273-4a2b-a234-951b1d50f056" (UID: "f7f09fbf-5273-4a2b-a234-951b1d50f056"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.499225 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.499113    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wptgp\" (UniqueName: \"kubernetes.io/projected/f7f09fbf-5273-4a2b-a234-951b1d50f056-kube-api-access-wptgp\") pod \"f7f09fbf-5273-4a2b-a234-951b1d50f056\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") "
Sep 16 20:13:40.499225 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.499151    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-dshm\") pod \"f7f09fbf-5273-4a2b-a234-951b1d50f056\" (UID: \"f7f09fbf-5273-4a2b-a234-951b1d50f056\") "
Sep 16 20:13:40.499432 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.499273    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7f09fbf-5273-4a2b-a234-951b1d50f056" (UID: "f7f09fbf-5273-4a2b-a234-951b1d50f056"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.499432 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.499374    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f09fbf-5273-4a2b-a234-951b1d50f056-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7f09fbf-5273-4a2b-a234-951b1d50f056" (UID: "f7f09fbf-5273-4a2b-a234-951b1d50f056"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:13:40.499543 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.499476    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.499543 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.499495    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.499632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.499603    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-home" (OuterVolumeSpecName: "home") pod "f7f09fbf-5273-4a2b-a234-951b1d50f056" (UID: "f7f09fbf-5273-4a2b-a234-951b1d50f056"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.501486 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.501456    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-dshm" (OuterVolumeSpecName: "dshm") pod "f7f09fbf-5273-4a2b-a234-951b1d50f056" (UID: "f7f09fbf-5273-4a2b-a234-951b1d50f056"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.501869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.501845    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f09fbf-5273-4a2b-a234-951b1d50f056-kube-api-access-wptgp" (OuterVolumeSpecName: "kube-api-access-wptgp") pod "f7f09fbf-5273-4a2b-a234-951b1d50f056" (UID: "f7f09fbf-5273-4a2b-a234-951b1d50f056"). InnerVolumeSpecName "kube-api-access-wptgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:13:40.501950 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.501897    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f09fbf-5273-4a2b-a234-951b1d50f056-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7f09fbf-5273-4a2b-a234-951b1d50f056" (UID: "f7f09fbf-5273-4a2b-a234-951b1d50f056"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:13:40.518867 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.518835    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7f09fbf-5273-4a2b-a234-951b1d50f056" (UID: "f7f09fbf-5273-4a2b-a234-951b1d50f056"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:13:40.600892 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.600865    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.600892 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.600890    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.601063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.600900    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f09fbf-5273-4a2b-a234-951b1d50f056-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.601063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.600911    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wptgp\" (UniqueName: \"kubernetes.io/projected/f7f09fbf-5273-4a2b-a234-951b1d50f056-kube-api-access-wptgp\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.601063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.600918    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f09fbf-5273-4a2b-a234-951b1d50f056-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:13:40.601063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:40.600926    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7f09fbf-5273-4a2b-a234-951b1d50f056-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:13:41.424627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:41.424550    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"
Sep 16 20:13:41.424627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:41.424572    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52" event={"ID":"f7f09fbf-5273-4a2b-a234-951b1d50f056","Type":"ContainerDied","Data":"aea1412de5a6f64580ead8c90335ee7ce36a555fbbe6ae241cf02e3128902f3e"}
Sep 16 20:13:41.425101 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:41.424640    2574 scope.go:117] "RemoveContainer" containerID="7a7803946189c44dbcc406063370acef81f923dfe194a1def6797de7af259cab"
Sep 16 20:13:41.433864 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:41.433840    2574 scope.go:117] "RemoveContainer" containerID="9f3ce2a038f46150dc4556b3efb7874207f67c9a8bd3a0c70be5b36b26ef9608"
Sep 16 20:13:41.448836 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:41.448727    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"]
Sep 16 20:13:41.455637 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:41.455610    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-79db8fd4b6-9rz52"]
Sep 16 20:13:43.088883 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:43.088849    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f09fbf-5273-4a2b-a234-951b1d50f056" path="/var/lib/kubelet/pods/f7f09fbf-5273-4a2b-a234-951b1d50f056/volumes"
Sep 16 20:13:48.690561 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:13:48.690524    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:10.147614 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:10.147526    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:10.250067 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:10.250021    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:10.348268 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:10.348231    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:14.486940 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:14.486897    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:15.221123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:15.221087    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:28.640623 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:28.640574    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:28.734251 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:28.734207    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:29.053843 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:29.053753    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:14:29.054009 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:29.053894    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:14:32.944170 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:32.944132    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:35.778548 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:35.778517    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:41.654282 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:41.654244    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:41.711652 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:41.711612    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:43.930339 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:43.930306    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:52.342150 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:52.342110    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:58.123233 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:58.123184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:14:58.186930 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:14:58.186891    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:01.049089 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.049055    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:01.913493 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.913456    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"]
Sep 16 20:15:01.913793 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.913781    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerName="main"
Sep 16 20:15:01.913848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.913796    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerName="main"
Sep 16 20:15:01.913848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.913813    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerName="storage-initializer"
Sep 16 20:15:01.913848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.913820    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerName="storage-initializer"
Sep 16 20:15:01.913950 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.913890    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f09fbf-5273-4a2b-a234-951b1d50f056" containerName="main"
Sep 16 20:15:01.916859 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.916844    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:01.919726 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.919707    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 20:15:01.920385 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.920368    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 20:15:01.921373 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.921352    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-bn8cc\""
Sep 16 20:15:01.921535 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.921352    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:15:01.921535 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.921358    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 20:15:01.921535 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.921361    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 20:15:01.929759 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.929738    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"]
Sep 16 20:15:01.934767 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:01.934730    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:02.104653 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.104610    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.104653 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.104657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrpww\" (UniqueName: \"kubernetes.io/projected/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kube-api-access-lrpww\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.105087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.104687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-dshm\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.105087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.104721    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-home\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.105087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.104743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/284f80ea-6e89-42d6-9840-ec2cb91d6e46-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.105087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.104790    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tls-certs\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.105087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.104829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-model-cache\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.105087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.104843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.186309 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.186209    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:02.206293 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/284f80ea-6e89-42d6-9840-ec2cb91d6e46-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.206504 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tls-certs\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.206504 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-model-cache\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.206504 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206362    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.206504 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.206504 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpww\" (UniqueName: \"kubernetes.io/projected/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kube-api-access-lrpww\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.206504 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-dshm\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.206504 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206495    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-home\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.206996 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206833    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-model-cache\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.206996 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.206903    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tmp-dir\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.207325 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.207110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.207325 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.207135    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-home\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.207325 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.207175    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/284f80ea-6e89-42d6-9840-ec2cb91d6e46-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.208897 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.208876    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-dshm\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.209245 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.209227    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tls-certs\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.214348 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.214326    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrpww\" (UniqueName: \"kubernetes.io/projected/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kube-api-access-lrpww\") pod \"tokenizer-template-args-test-kserve-54d4855974-x5j5l\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.227419 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.227378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:02.296875 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.296836    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:02.504689 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.504475    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"]
Sep 16 20:15:02.507559 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:15:02.507532    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284f80ea_6e89_42d6_9840_ec2cb91d6e46.slice/crio-7ab2fd7ded9be9c73344fe8fda00abaa322c2e338ee41d81178ec3554bbcc08c WatchSource:0}: Error finding container 7ab2fd7ded9be9c73344fe8fda00abaa322c2e338ee41d81178ec3554bbcc08c: Status 404 returned error can't find the container with id 7ab2fd7ded9be9c73344fe8fda00abaa322c2e338ee41d81178ec3554bbcc08c
Sep 16 20:15:02.854205 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.854162    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" event={"ID":"284f80ea-6e89-42d6-9840-ec2cb91d6e46","Type":"ContainerStarted","Data":"a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d"}
Sep 16 20:15:02.854385 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:02.854236    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" event={"ID":"284f80ea-6e89-42d6-9840-ec2cb91d6e46","Type":"ContainerStarted","Data":"7ab2fd7ded9be9c73344fe8fda00abaa322c2e338ee41d81178ec3554bbcc08c"}
Sep 16 20:15:05.874924 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:05.874887    2574 generic.go:356] "Generic (PLEG): container finished" podID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" containerID="a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d" exitCode=0
Sep 16 20:15:05.875432 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:05.874943    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" event={"ID":"284f80ea-6e89-42d6-9840-ec2cb91d6e46","Type":"ContainerDied","Data":"a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d"}
Sep 16 20:15:06.881873 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:06.881831    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" event={"ID":"284f80ea-6e89-42d6-9840-ec2cb91d6e46","Type":"ContainerStarted","Data":"e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02"}
Sep 16 20:15:07.228324 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:07.228236    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:12.228477 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:12.228439    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:12.241306 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:12.241277    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:12.268189 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:12.268120    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" podStartSLOduration=8.992270282 podStartE2EDuration="11.268098323s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.275828041s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:15:06.910660216 +0000 UTC m=+2138.468005348" watchObservedRunningTime="2026-09-16 20:15:12.268098323 +0000 UTC m=+2143.825443412"
Sep 16 20:15:12.922345 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:12.922312    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:20.512464 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:20.512430    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:23.560967 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:23.560877    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:37.899302 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:37.899261    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:40.869481 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:40.869438    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:40.946081 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:40.946036    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"]
Sep 16 20:15:40.955517 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:15:40.955477    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:15:40.955517 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:15:40.955517 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:15:40.955517 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02" cmd=["/bin/sleep","15"]
Sep 16 20:15:40.955767 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:40.955537    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" message="" err=<
Sep 16 20:15:40.955767 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:15:40.955767 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:15:40.955767 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:15:40.955767 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:15:40.955549    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:15:40.955767 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:15:40.955767 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:15:40.955767 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" podUID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" containerName="main" containerID="cri-o://e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02"
Sep 16 20:15:40.955767 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:40.955584    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" podUID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" containerName="main" containerID="cri-o://e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02" gracePeriod=60
Sep 16 20:15:40.971356 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:40.971323    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:41.045525 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.045472    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:41.311805 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.311774    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:41.447162 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447063    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/284f80ea-6e89-42d6-9840-ec2cb91d6e46-cabundle-cert\") pod \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") "
Sep 16 20:15:41.447162 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447113    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrpww\" (UniqueName: \"kubernetes.io/projected/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kube-api-access-lrpww\") pod \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") "
Sep 16 20:15:41.447162 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447136    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-dshm\") pod \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") "
Sep 16 20:15:41.447848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447265    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tmp-dir\") pod \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") "
Sep 16 20:15:41.447848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447307    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tls-certs\") pod \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") "
Sep 16 20:15:41.447848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447347    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-home\") pod \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") "
Sep 16 20:15:41.447848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447369    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-model-cache\") pod \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") "
Sep 16 20:15:41.447848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447398    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kserve-provision-location\") pod \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\" (UID: \"284f80ea-6e89-42d6-9840-ec2cb91d6e46\") "
Sep 16 20:15:41.447848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447507    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284f80ea-6e89-42d6-9840-ec2cb91d6e46-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "284f80ea-6e89-42d6-9840-ec2cb91d6e46" (UID: "284f80ea-6e89-42d6-9840-ec2cb91d6e46"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:15:41.447848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447683    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-model-cache" (OuterVolumeSpecName: "model-cache") pod "284f80ea-6e89-42d6-9840-ec2cb91d6e46" (UID: "284f80ea-6e89-42d6-9840-ec2cb91d6e46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.448247 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447793    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-home" (OuterVolumeSpecName: "home") pod "284f80ea-6e89-42d6-9840-ec2cb91d6e46" (UID: "284f80ea-6e89-42d6-9840-ec2cb91d6e46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.448247 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447851    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "284f80ea-6e89-42d6-9840-ec2cb91d6e46" (UID: "284f80ea-6e89-42d6-9840-ec2cb91d6e46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.448247 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447817    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.448247 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.447969    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/284f80ea-6e89-42d6-9840-ec2cb91d6e46-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.449519 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.449475    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-dshm" (OuterVolumeSpecName: "dshm") pod "284f80ea-6e89-42d6-9840-ec2cb91d6e46" (UID: "284f80ea-6e89-42d6-9840-ec2cb91d6e46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.449616 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.449562    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kube-api-access-lrpww" (OuterVolumeSpecName: "kube-api-access-lrpww") pod "284f80ea-6e89-42d6-9840-ec2cb91d6e46" (UID: "284f80ea-6e89-42d6-9840-ec2cb91d6e46"). InnerVolumeSpecName "kube-api-access-lrpww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:15:41.449616 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.449575    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "284f80ea-6e89-42d6-9840-ec2cb91d6e46" (UID: "284f80ea-6e89-42d6-9840-ec2cb91d6e46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:15:41.468448 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.468397    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "284f80ea-6e89-42d6-9840-ec2cb91d6e46" (UID: "284f80ea-6e89-42d6-9840-ec2cb91d6e46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:15:41.549220 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.549178    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrpww\" (UniqueName: \"kubernetes.io/projected/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kube-api-access-lrpww\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.549220 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.549212    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.549220 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.549222    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.549519 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.549240    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284f80ea-6e89-42d6-9840-ec2cb91d6e46-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.549519 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.549250    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:15:41.549519 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:41.549257    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/284f80ea-6e89-42d6-9840-ec2cb91d6e46-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:15:42.064439 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.064388    2574 generic.go:356] "Generic (PLEG): container finished" podID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" containerID="e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02" exitCode=0
Sep 16 20:15:42.064827 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.064454    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"
Sep 16 20:15:42.064827 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.064477    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" event={"ID":"284f80ea-6e89-42d6-9840-ec2cb91d6e46","Type":"ContainerDied","Data":"e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02"}
Sep 16 20:15:42.064827 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.064536    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l" event={"ID":"284f80ea-6e89-42d6-9840-ec2cb91d6e46","Type":"ContainerDied","Data":"7ab2fd7ded9be9c73344fe8fda00abaa322c2e338ee41d81178ec3554bbcc08c"}
Sep 16 20:15:42.064827 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.064563    2574 scope.go:117] "RemoveContainer" containerID="e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02"
Sep 16 20:15:42.075068 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.075047    2574 scope.go:117] "RemoveContainer" containerID="a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d"
Sep 16 20:15:42.083429 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.083397    2574 scope.go:117] "RemoveContainer" containerID="e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02"
Sep 16 20:15:42.083724 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:15:42.083703    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02\": container with ID starting with e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02 not found: ID does not exist" containerID="e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02"
Sep 16 20:15:42.083776 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.083734    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02"} err="failed to get container status \"e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02\": rpc error: code = NotFound desc = could not find container \"e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02\": container with ID starting with e77073b41ef8986ff425f9a17c254bc6268b2dcfc2ab6f9dfbddfeb906887f02 not found: ID does not exist"
Sep 16 20:15:42.083776 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.083753    2574 scope.go:117] "RemoveContainer" containerID="a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d"
Sep 16 20:15:42.084015 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:15:42.083999    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d\": container with ID starting with a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d not found: ID does not exist" containerID="a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d"
Sep 16 20:15:42.084070 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.084017    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d"} err="failed to get container status \"a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d\": rpc error: code = NotFound desc = could not find container \"a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d\": container with ID starting with a84cb74e03a85d93fee98753c17af6a0050241ced40d7c31de9e4452a619272d not found: ID does not exist"
Sep 16 20:15:42.088745 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.088698    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"]
Sep 16 20:15:42.091231 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:42.091212    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-54d4855974-x5j5l"]
Sep 16 20:15:43.088136 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:43.088101    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" path="/var/lib/kubelet/pods/284f80ea-6e89-42d6-9840-ec2cb91d6e46/volumes"
Sep 16 20:15:53.546761 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:53.546721    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:53.638619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:53.638585    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:53.722216 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:53.722183    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:15:58.613327 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:15:58.613286    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:13.114674 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:13.114638    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:30.823279 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:30.823237    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:30.946147 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:30.946108    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:33.821271 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:33.821229    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:36.892810 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:36.892767    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:38.266869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:38.266833    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:42.843753 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:42.843724    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:42.914342 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:42.914308    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:44.415741 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:44.415698    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:16:44.515319 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:16:44.515276    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:00.814959 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:00.814865    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:01.114286 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:01.114244    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:23.221858 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:23.221824    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:33.465020 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:33.464979    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:33.624333 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:33.624284    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:33.725598 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:33.725500    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:35.023965 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:35.023928    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:35.788303 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:35.788253    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:38.548926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:38.548877    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:49.767942 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:49.767908    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:49.838081 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:49.838044    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:56.895116 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.895083    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"]
Sep 16 20:17:56.895748 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.895430    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" containerName="storage-initializer"
Sep 16 20:17:56.895748 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.895443    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" containerName="storage-initializer"
Sep 16 20:17:56.895748 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.895452    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" containerName="main"
Sep 16 20:17:56.895748 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.895458    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" containerName="main"
Sep 16 20:17:56.895748 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.895525    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="284f80ea-6e89-42d6-9840-ec2cb91d6e46" containerName="main"
Sep 16 20:17:56.897391 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.897375    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:56.901123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.901097    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-slm8b\""
Sep 16 20:17:56.901123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.901110    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 20:17:56.901342 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.901125    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 20:17:56.901342 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.901132    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 16 20:17:56.909670 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.909647    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"]
Sep 16 20:17:56.940600 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.940565    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:17:56.987545 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.987509    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:56.987733 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.987557    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnh82\" (UniqueName: \"kubernetes.io/projected/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-kube-api-access-nnh82\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:56.987733 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.987648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:56.987733 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.987692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:56.987863 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.987733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-dshm\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:56.987863 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:56.987776    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-home\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.088249 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.088212    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.088478 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.088263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnh82\" (UniqueName: \"kubernetes.io/projected/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-kube-api-access-nnh82\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.088478 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.088300    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.088478 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.088332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.088478 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.088368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-dshm\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.088701 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.088516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-home\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.088883 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.088729    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.088883 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.088763    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.088989 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.088908    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-home\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.090751 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.090730    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-dshm\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.091020 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.090991    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.096287 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.096258    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnh82\" (UniqueName: \"kubernetes.io/projected/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-kube-api-access-nnh82\") pod \"tri-v1-kserve-79f8f98bc5-mvs6q\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.211217 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.211132    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:17:57.494624 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.494593    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"]
Sep 16 20:17:57.496044 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:17:57.496017    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bf1359_a4ef_4d3d_8a34_814b2791fdb6.slice/crio-c5bbb25fd00cedd900ca7302c634dacb24a4b820ffcda72723632ee051dc72f2 WatchSource:0}: Error finding container c5bbb25fd00cedd900ca7302c634dacb24a4b820ffcda72723632ee051dc72f2: Status 404 returned error can't find the container with id c5bbb25fd00cedd900ca7302c634dacb24a4b820ffcda72723632ee051dc72f2
Sep 16 20:17:57.772787 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.772689    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q" event={"ID":"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6","Type":"ContainerStarted","Data":"a0fabc4e6b662be36a0bd633cbcded355db13a9f88afa6537bbc23ba83f6ddfb"}
Sep 16 20:17:57.772939 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.772786    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q" event={"ID":"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6","Type":"ContainerStarted","Data":"c5bbb25fd00cedd900ca7302c634dacb24a4b820ffcda72723632ee051dc72f2"}
Sep 16 20:17:57.794902 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:57.794844    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q" podStartSLOduration=1.79482965 podStartE2EDuration="1.79482965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:17:57.792867191 +0000 UTC m=+2309.350212276" watchObservedRunningTime="2026-09-16 20:17:57.79482965 +0000 UTC m=+2309.352174788"
Sep 16 20:17:58.211470 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:17:58.211431    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:18:04.679169 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.679128    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"]
Sep 16 20:18:04.682541 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.682517    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.685155 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.685128    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 16 20:18:04.695993 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.695964    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"]
Sep 16 20:18:04.726896 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.726867    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:04.752815 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.752782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.752815 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.752817    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-dshm\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.753040 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.752850    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-home\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.753040 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.752909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-model-cache\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.753040 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.752950    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvdh\" (UniqueName: \"kubernetes.io/projected/90d0e015-35fc-47c2-bcb3-8f19062b62df-kube-api-access-6bvdh\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.753040 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.753021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d0e015-35fc-47c2-bcb3-8f19062b62df-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.854061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.854020    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d0e015-35fc-47c2-bcb3-8f19062b62df-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.854270 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.854091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.854270 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.854118    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-dshm\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.854270 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.854154    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-home\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.854270 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.854175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-model-cache\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.854270 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.854198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvdh\" (UniqueName: \"kubernetes.io/projected/90d0e015-35fc-47c2-bcb3-8f19062b62df-kube-api-access-6bvdh\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.854664 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.854637    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-home\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.854777 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.854690    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-model-cache\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.854777 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.854703    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.856699 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.856676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-dshm\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.856798 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.856754    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d0e015-35fc-47c2-bcb3-8f19062b62df-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.863035 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.863010    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvdh\" (UniqueName: \"kubernetes.io/projected/90d0e015-35fc-47c2-bcb3-8f19062b62df-kube-api-access-6bvdh\") pod \"tri-v2-kserve-dcb554bf5-d7bdj\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:04.995136 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:04.995046    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:05.273454 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:05.273429    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"]
Sep 16 20:18:05.274832 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:18:05.274801    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d0e015_35fc_47c2_bcb3_8f19062b62df.slice/crio-4994d80eb0f62504056fc2afd15d720af1556335d8c1ae2e1ae6780c2aceb185 WatchSource:0}: Error finding container 4994d80eb0f62504056fc2afd15d720af1556335d8c1ae2e1ae6780c2aceb185: Status 404 returned error can't find the container with id 4994d80eb0f62504056fc2afd15d720af1556335d8c1ae2e1ae6780c2aceb185
Sep 16 20:18:05.816507 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:05.816469    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj" event={"ID":"90d0e015-35fc-47c2-bcb3-8f19062b62df","Type":"ContainerStarted","Data":"9c34969b13a552166220a7dcef27c0844aa510bbffd2724245aca4a3532e6532"}
Sep 16 20:18:05.816906 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:05.816526    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj" event={"ID":"90d0e015-35fc-47c2-bcb3-8f19062b62df","Type":"ContainerStarted","Data":"4994d80eb0f62504056fc2afd15d720af1556335d8c1ae2e1ae6780c2aceb185"}
Sep 16 20:18:05.839560 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:05.839503    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj" podStartSLOduration=1.839481999 podStartE2EDuration="1.839481999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:18:05.839100358 +0000 UTC m=+2317.396445456" watchObservedRunningTime="2026-09-16 20:18:05.839481999 +0000 UTC m=+2317.396827087"
Sep 16 20:18:05.995865 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:05.995832    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:06.056359 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:06.056320    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:06.122862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:06.122829    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:07.211467 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:07.211430    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:18:07.224252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:07.224227    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:18:07.837832 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:07.837802    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:18:14.995555 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:14.995520    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:15.008165 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:15.008134    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:15.784482 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:15.784440    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:15.879389 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:15.879357    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:18:19.334372 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:19.334339    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:19.393755 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:19.393715    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:21.674603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:21.674521    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:21.751735 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:21.751692    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:46.518431 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:46.518385    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:56.367147 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:56.367092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:18:56.465309 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:18:56.465276    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:19:02.807959 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:02.807918    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:19:08.824734 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:08.824697    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:19:08.893349 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:08.893321    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:19:29.089852 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:29.089820    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:19:29.091770 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:29.091747    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:19:30.287577 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:30.287541    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:19:30.592749 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:30.592716    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:19:39.324699 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:39.324658    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:19:50.408719 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:50.408677    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:19:50.474059 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:19:50.474027    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:14.001863 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:14.001776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:23.462277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.462231    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:23.840119 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.840084    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:23.877716 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.877682    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"]
Sep 16 20:20:23.880470 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.880453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:23.883128 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.883106    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 20:20:23.884042 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.884016    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 20:20:23.884141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.884016    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 20:20:23.884141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.884039    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-lhn98\""
Sep 16 20:20:23.890007 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.889984    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"]
Sep 16 20:20:23.913112 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.913071    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:23.978225 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.978194    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:23.992303 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.992273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:23.992303 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.992304    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:23.992527 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.992327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd78a1b4-e893-4254-8410-58f6313d7ea1-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:23.992527 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.992349    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcsdd\" (UniqueName: \"kubernetes.io/projected/fd78a1b4-e893-4254-8410-58f6313d7ea1-kube-api-access-mcsdd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:23.992527 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.992470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:23.992527 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:23.992517    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.093889 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.093803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd78a1b4-e893-4254-8410-58f6313d7ea1-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.093889 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.093846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsdd\" (UniqueName: \"kubernetes.io/projected/fd78a1b4-e893-4254-8410-58f6313d7ea1-kube-api-access-mcsdd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.094125 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.093904    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.094125 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.093937    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.094125 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.093979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.094125 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.093998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.094479 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.094455    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.094545 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.094495    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.094545 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.094460    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.096251 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.096221    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.096353 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.096321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd78a1b4-e893-4254-8410-58f6313d7ea1-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.102089 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.102064    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsdd\" (UniqueName: \"kubernetes.io/projected/fd78a1b4-e893-4254-8410-58f6313d7ea1-kube-api-access-mcsdd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.191909 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.191867    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:24.291610 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.291569    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:24.465372 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.465345    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"]
Sep 16 20:20:24.467183 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:20:24.467151    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd78a1b4_e893_4254_8410_58f6313d7ea1.slice/crio-b50d1a353fd73ae4a455fc79d20b857af7f2aabe3803a62e81e4e8e95c32db28 WatchSource:0}: Error finding container b50d1a353fd73ae4a455fc79d20b857af7f2aabe3803a62e81e4e8e95c32db28: Status 404 returned error can't find the container with id b50d1a353fd73ae4a455fc79d20b857af7f2aabe3803a62e81e4e8e95c32db28
Sep 16 20:20:24.564825 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.564780    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn" event={"ID":"fd78a1b4-e893-4254-8410-58f6313d7ea1","Type":"ContainerStarted","Data":"314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2"}
Sep 16 20:20:24.565009 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.564857    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn" event={"ID":"fd78a1b4-e893-4254-8410-58f6313d7ea1","Type":"ContainerStarted","Data":"b50d1a353fd73ae4a455fc79d20b857af7f2aabe3803a62e81e4e8e95c32db28"}
Sep 16 20:20:24.585364 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:24.585317    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn" podStartSLOduration=1.585299384 podStartE2EDuration="1.585299384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:20:24.584341526 +0000 UTC m=+2456.141686642" watchObservedRunningTime="2026-09-16 20:20:24.585299384 +0000 UTC m=+2456.142644876"
Sep 16 20:20:25.192878 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:25.192841    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:28.446664 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:28.446623    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"]
Sep 16 20:20:28.451923 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:28.451890    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"]
Sep 16 20:20:28.520093 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:28.520055    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:28.795074 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:28.794981    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:29.577018 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:29.576980    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:30.087502 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:30.087463    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:34.192181 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:34.192143    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:34.205330 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:34.205306    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:34.629544 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:34.629513    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:20:37.592332 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:37.592293    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:43.468649 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.468584    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q" podUID="b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" containerName="main" containerID="cri-o://a0fabc4e6b662be36a0bd633cbcded355db13a9f88afa6537bbc23ba83f6ddfb" gracePeriod=45
Sep 16 20:20:43.472538 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.472489    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj" podUID="90d0e015-35fc-47c2-bcb3-8f19062b62df" containerName="main" containerID="cri-o://9c34969b13a552166220a7dcef27c0844aa510bbffd2724245aca4a3532e6532" gracePeriod=45
Sep 16 20:20:43.668929 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.668895    2574 generic.go:356] "Generic (PLEG): container finished" podID="b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" containerID="a0fabc4e6b662be36a0bd633cbcded355db13a9f88afa6537bbc23ba83f6ddfb" exitCode=0
Sep 16 20:20:43.669096 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.668965    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q" event={"ID":"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6","Type":"ContainerDied","Data":"a0fabc4e6b662be36a0bd633cbcded355db13a9f88afa6537bbc23ba83f6ddfb"}
Sep 16 20:20:43.670829 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.670806    2574 generic.go:356] "Generic (PLEG): container finished" podID="90d0e015-35fc-47c2-bcb3-8f19062b62df" containerID="9c34969b13a552166220a7dcef27c0844aa510bbffd2724245aca4a3532e6532" exitCode=0
Sep 16 20:20:43.670951 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.670882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj" event={"ID":"90d0e015-35fc-47c2-bcb3-8f19062b62df","Type":"ContainerDied","Data":"9c34969b13a552166220a7dcef27c0844aa510bbffd2724245aca4a3532e6532"}
Sep 16 20:20:43.736681 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.736622    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:20:43.754653 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.754632    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:20:43.865937 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.865906    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-tmp-dir\") pod \"90d0e015-35fc-47c2-bcb3-8f19062b62df\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") "
Sep 16 20:20:43.865937 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.865943    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d0e015-35fc-47c2-bcb3-8f19062b62df-tls-certs\") pod \"90d0e015-35fc-47c2-bcb3-8f19062b62df\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") "
Sep 16 20:20:43.866178 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.865971    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-model-cache\") pod \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") "
Sep 16 20:20:43.866178 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.865998    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tmp-dir\") pod \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") "
Sep 16 20:20:43.866178 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866031    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-home\") pod \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") "
Sep 16 20:20:43.866178 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866078    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnh82\" (UniqueName: \"kubernetes.io/projected/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-kube-api-access-nnh82\") pod \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") "
Sep 16 20:20:43.866178 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866101    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bvdh\" (UniqueName: \"kubernetes.io/projected/90d0e015-35fc-47c2-bcb3-8f19062b62df-kube-api-access-6bvdh\") pod \"90d0e015-35fc-47c2-bcb3-8f19062b62df\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") "
Sep 16 20:20:43.866178 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866145    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-dshm\") pod \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") "
Sep 16 20:20:43.866178 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866175    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-model-cache\") pod \"90d0e015-35fc-47c2-bcb3-8f19062b62df\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") "
Sep 16 20:20:43.866586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866208    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-home\") pod \"90d0e015-35fc-47c2-bcb3-8f19062b62df\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") "
Sep 16 20:20:43.866586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866244    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-dshm\") pod \"90d0e015-35fc-47c2-bcb3-8f19062b62df\" (UID: \"90d0e015-35fc-47c2-bcb3-8f19062b62df\") "
Sep 16 20:20:43.866586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866282    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tls-certs\") pod \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\" (UID: \"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6\") "
Sep 16 20:20:43.866586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866286    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" (UID: "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.866586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866332    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-home" (OuterVolumeSpecName: "home") pod "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" (UID: "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.866586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866343    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" (UID: "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.866586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866545    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-model-cache" (OuterVolumeSpecName: "model-cache") pod "90d0e015-35fc-47c2-bcb3-8f19062b62df" (UID: "90d0e015-35fc-47c2-bcb3-8f19062b62df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.866926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866635    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90d0e015-35fc-47c2-bcb3-8f19062b62df" (UID: "90d0e015-35fc-47c2-bcb3-8f19062b62df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.866926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866654    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-home" (OuterVolumeSpecName: "home") pod "90d0e015-35fc-47c2-bcb3-8f19062b62df" (UID: "90d0e015-35fc-47c2-bcb3-8f19062b62df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.866926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866717    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.866926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866735    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.866926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866749    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.866926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866762    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.866926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.866775    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.868563 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.868539    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d0e015-35fc-47c2-bcb3-8f19062b62df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90d0e015-35fc-47c2-bcb3-8f19062b62df" (UID: "90d0e015-35fc-47c2-bcb3-8f19062b62df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:20:43.868685 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.868539    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" (UID: "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:20:43.868685 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.868650    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-dshm" (OuterVolumeSpecName: "dshm") pod "90d0e015-35fc-47c2-bcb3-8f19062b62df" (UID: "90d0e015-35fc-47c2-bcb3-8f19062b62df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.868685 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.868653    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-kube-api-access-nnh82" (OuterVolumeSpecName: "kube-api-access-nnh82") pod "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" (UID: "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6"). InnerVolumeSpecName "kube-api-access-nnh82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:20:43.869001 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.868982    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-dshm" (OuterVolumeSpecName: "dshm") pod "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" (UID: "b2bf1359-a4ef-4d3d-8a34-814b2791fdb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:20:43.869113 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.868998    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d0e015-35fc-47c2-bcb3-8f19062b62df-kube-api-access-6bvdh" (OuterVolumeSpecName: "kube-api-access-6bvdh") pod "90d0e015-35fc-47c2-bcb3-8f19062b62df" (UID: "90d0e015-35fc-47c2-bcb3-8f19062b62df"). InnerVolumeSpecName "kube-api-access-6bvdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:20:43.967632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.967599    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.967632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.967626    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d0e015-35fc-47c2-bcb3-8f19062b62df-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.967632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.967636    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.967869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.967646    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d0e015-35fc-47c2-bcb3-8f19062b62df-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.967869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.967655    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnh82\" (UniqueName: \"kubernetes.io/projected/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-kube-api-access-nnh82\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.967869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.967665    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvdh\" (UniqueName: \"kubernetes.io/projected/90d0e015-35fc-47c2-bcb3-8f19062b62df-kube-api-access-6bvdh\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:43.967869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:43.967674    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:20:44.677558 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.677527    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"
Sep 16 20:20:44.678000 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.677568    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj" event={"ID":"90d0e015-35fc-47c2-bcb3-8f19062b62df","Type":"ContainerDied","Data":"4994d80eb0f62504056fc2afd15d720af1556335d8c1ae2e1ae6780c2aceb185"}
Sep 16 20:20:44.678000 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.677633    2574 scope.go:117] "RemoveContainer" containerID="9c34969b13a552166220a7dcef27c0844aa510bbffd2724245aca4a3532e6532"
Sep 16 20:20:44.679004 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.678932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q" event={"ID":"b2bf1359-a4ef-4d3d-8a34-814b2791fdb6","Type":"ContainerDied","Data":"c5bbb25fd00cedd900ca7302c634dacb24a4b820ffcda72723632ee051dc72f2"}
Sep 16 20:20:44.679071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.679010    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"
Sep 16 20:20:44.687335 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.687319    2574 scope.go:117] "RemoveContainer" containerID="a0fabc4e6b662be36a0bd633cbcded355db13a9f88afa6537bbc23ba83f6ddfb"
Sep 16 20:20:44.700745 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.700720    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"]
Sep 16 20:20:44.704761 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.704739    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d7bdj"]
Sep 16 20:20:44.714347 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.714317    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"]
Sep 16 20:20:44.717666 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:44.717646    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-mvs6q"]
Sep 16 20:20:45.088267 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:45.088236    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d0e015-35fc-47c2-bcb3-8f19062b62df" path="/var/lib/kubelet/pods/90d0e015-35fc-47c2-bcb3-8f19062b62df/volumes"
Sep 16 20:20:45.088726 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:45.088710    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" path="/var/lib/kubelet/pods/b2bf1359-a4ef-4d3d-8a34-814b2791fdb6/volumes"
Sep 16 20:20:52.186496 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.186464    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"]
Sep 16 20:20:52.186850 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.186833    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d0e015-35fc-47c2-bcb3-8f19062b62df" containerName="main"
Sep 16 20:20:52.186850 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.186848    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d0e015-35fc-47c2-bcb3-8f19062b62df" containerName="main"
Sep 16 20:20:52.186923 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.186868    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" containerName="main"
Sep 16 20:20:52.186923 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.186873    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" containerName="main"
Sep 16 20:20:52.186985 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.186933    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d0e015-35fc-47c2-bcb3-8f19062b62df" containerName="main"
Sep 16 20:20:52.186985 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.186945    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2bf1359-a4ef-4d3d-8a34-814b2791fdb6" containerName="main"
Sep 16 20:20:52.189889 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.189867    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.193512 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.193487    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 20:20:52.193512 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.193487    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 20:20:52.193697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.193526    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-99pbq\""
Sep 16 20:20:52.193697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.193585    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 20:20:52.200809 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.200780    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"]
Sep 16 20:20:52.216788 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.216747    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:20:52.236441 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.236390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.236606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.236455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-model-cache\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.236606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.236488    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66xm\" (UniqueName: \"kubernetes.io/projected/dad22229-e045-46d9-96dc-eda72f8e49df-kube-api-access-f66xm\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.236606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.236550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad22229-e045-46d9-96dc-eda72f8e49df-tls-certs\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.236606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.236579    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-home\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.236606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.236596    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-dshm\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.337127 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.337088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad22229-e045-46d9-96dc-eda72f8e49df-tls-certs\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.337323 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.337135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-home\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.337323 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.337159    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-dshm\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.337323 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.337216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.337323 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.337247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-model-cache\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.337573 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.337376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66xm\" (UniqueName: \"kubernetes.io/projected/dad22229-e045-46d9-96dc-eda72f8e49df-kube-api-access-f66xm\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.338139 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.338108    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-home\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.338296 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.338145    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-model-cache\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.338296 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.338176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.340795 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.340425    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-dshm\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.346122 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.345090    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad22229-e045-46d9-96dc-eda72f8e49df-tls-certs\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.347221 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.347197    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66xm\" (UniqueName: \"kubernetes.io/projected/dad22229-e045-46d9-96dc-eda72f8e49df-kube-api-access-f66xm\") pod \"late-v1-kserve-56cbc8456c-xzx5h\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.503077 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.502982    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:20:52.778776 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:52.778750    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"]
Sep 16 20:20:52.780165 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:20:52.780132    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad22229_e045_46d9_96dc_eda72f8e49df.slice/crio-c320e81105c9587056a9b9072c71af92547e23358a3cc5854ccb0a1b31ede9fb WatchSource:0}: Error finding container c320e81105c9587056a9b9072c71af92547e23358a3cc5854ccb0a1b31ede9fb: Status 404 returned error can't find the container with id c320e81105c9587056a9b9072c71af92547e23358a3cc5854ccb0a1b31ede9fb
Sep 16 20:20:53.730668 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:53.730631    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h" event={"ID":"dad22229-e045-46d9-96dc-eda72f8e49df","Type":"ContainerStarted","Data":"7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd"}
Sep 16 20:20:53.731088 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:53.730688    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h" event={"ID":"dad22229-e045-46d9-96dc-eda72f8e49df","Type":"ContainerStarted","Data":"c320e81105c9587056a9b9072c71af92547e23358a3cc5854ccb0a1b31ede9fb"}
Sep 16 20:20:53.748507 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:53.748449    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h" podStartSLOduration=1.7484298379999998 podStartE2EDuration="1.748429838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:20:53.748342963 +0000 UTC m=+2485.305688076" watchObservedRunningTime="2026-09-16 20:20:53.748429838 +0000 UTC m=+2485.305774926"
Sep 16 20:20:54.503677 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:20:54.503641    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:21:02.503956 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:02.503918    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:21:02.516713 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:02.516684    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:21:02.789819 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:02.789732    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:21:05.869639 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:05.869591    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:21:12.090314 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:12.090272    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:21:12.179344 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:12.179312    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:21:13.036120 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.036079    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:21:13.057588 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.057554    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"]
Sep 16 20:21:13.067004 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:21:13.066961    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:21:13.067004 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:21:13.067004 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:21:13.067004 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2" cmd=["/bin/sleep","15"]
Sep 16 20:21:13.067300 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.067023    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn" message="" err=<
Sep 16 20:21:13.067300 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:21:13.067300 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:21:13.067300 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:21:13.067300 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:21:13.067040    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:21:13.067300 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:21:13.067300 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:21:13.067300 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn" podUID="fd78a1b4-e893-4254-8410-58f6313d7ea1" containerName="main" containerID="cri-o://314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2"
Sep 16 20:21:13.067300 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.067086    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn" podUID="fd78a1b4-e893-4254-8410-58f6313d7ea1" containerName="main" containerID="cri-o://314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2" gracePeriod=60
Sep 16 20:21:13.138368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.138331    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:21:13.619073 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.619028    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn" podUID="fd78a1b4-e893-4254-8410-58f6313d7ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 16 20:21:13.812628 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.812604    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:21:13.838652 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.838624    2574 generic.go:356] "Generic (PLEG): container finished" podID="fd78a1b4-e893-4254-8410-58f6313d7ea1" containerID="314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2" exitCode=0
Sep 16 20:21:13.838817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.838678    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"
Sep 16 20:21:13.838817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.838707    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn" event={"ID":"fd78a1b4-e893-4254-8410-58f6313d7ea1","Type":"ContainerDied","Data":"314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2"}
Sep 16 20:21:13.838817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.838770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn" event={"ID":"fd78a1b4-e893-4254-8410-58f6313d7ea1","Type":"ContainerDied","Data":"b50d1a353fd73ae4a455fc79d20b857af7f2aabe3803a62e81e4e8e95c32db28"}
Sep 16 20:21:13.838817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.838799    2574 scope.go:117] "RemoveContainer" containerID="314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2"
Sep 16 20:21:13.848722 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.848700    2574 scope.go:117] "RemoveContainer" containerID="314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2"
Sep 16 20:21:13.849026 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:21:13.849006    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2\": container with ID starting with 314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2 not found: ID does not exist" containerID="314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2"
Sep 16 20:21:13.849122 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.849032    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2"} err="failed to get container status \"314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2\": rpc error: code = NotFound desc = could not find container \"314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2\": container with ID starting with 314217609ed7d278f70c5c16434e33225799a3ccb05d9095cf7a57df59372ea2 not found: ID does not exist"
Sep 16 20:21:13.937294 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937208    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-tmp-dir\") pod \"fd78a1b4-e893-4254-8410-58f6313d7ea1\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") "
Sep 16 20:21:13.937294 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937253    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-home\") pod \"fd78a1b4-e893-4254-8410-58f6313d7ea1\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") "
Sep 16 20:21:13.937294 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937289    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcsdd\" (UniqueName: \"kubernetes.io/projected/fd78a1b4-e893-4254-8410-58f6313d7ea1-kube-api-access-mcsdd\") pod \"fd78a1b4-e893-4254-8410-58f6313d7ea1\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") "
Sep 16 20:21:13.937619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937314    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-dshm\") pod \"fd78a1b4-e893-4254-8410-58f6313d7ea1\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") "
Sep 16 20:21:13.937619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937423    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd78a1b4-e893-4254-8410-58f6313d7ea1-tls-certs\") pod \"fd78a1b4-e893-4254-8410-58f6313d7ea1\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") "
Sep 16 20:21:13.937619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937483    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-model-cache\") pod \"fd78a1b4-e893-4254-8410-58f6313d7ea1\" (UID: \"fd78a1b4-e893-4254-8410-58f6313d7ea1\") "
Sep 16 20:21:13.937619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937540    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd78a1b4-e893-4254-8410-58f6313d7ea1" (UID: "fd78a1b4-e893-4254-8410-58f6313d7ea1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:21:13.937619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937556    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-home" (OuterVolumeSpecName: "home") pod "fd78a1b4-e893-4254-8410-58f6313d7ea1" (UID: "fd78a1b4-e893-4254-8410-58f6313d7ea1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:21:13.937897 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937772    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd78a1b4-e893-4254-8410-58f6313d7ea1" (UID: "fd78a1b4-e893-4254-8410-58f6313d7ea1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:21:13.937897 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937781    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:21:13.937897 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.937810    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:21:13.939734 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.939704    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-dshm" (OuterVolumeSpecName: "dshm") pod "fd78a1b4-e893-4254-8410-58f6313d7ea1" (UID: "fd78a1b4-e893-4254-8410-58f6313d7ea1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:21:13.939822 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.939785    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd78a1b4-e893-4254-8410-58f6313d7ea1-kube-api-access-mcsdd" (OuterVolumeSpecName: "kube-api-access-mcsdd") pod "fd78a1b4-e893-4254-8410-58f6313d7ea1" (UID: "fd78a1b4-e893-4254-8410-58f6313d7ea1"). InnerVolumeSpecName "kube-api-access-mcsdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:21:13.939822 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:13.939776    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd78a1b4-e893-4254-8410-58f6313d7ea1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd78a1b4-e893-4254-8410-58f6313d7ea1" (UID: "fd78a1b4-e893-4254-8410-58f6313d7ea1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:21:14.038625 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:14.038589    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:21:14.038625 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:14.038619    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcsdd\" (UniqueName: \"kubernetes.io/projected/fd78a1b4-e893-4254-8410-58f6313d7ea1-kube-api-access-mcsdd\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:21:14.038625 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:14.038631    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd78a1b4-e893-4254-8410-58f6313d7ea1-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:21:14.038869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:14.038641    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd78a1b4-e893-4254-8410-58f6313d7ea1-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:21:14.163333 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:14.163301    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"]
Sep 16 20:21:14.168512 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:14.168485    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995hv9nn"]
Sep 16 20:21:15.088304 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:15.088261    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd78a1b4-e893-4254-8410-58f6313d7ea1" path="/var/lib/kubelet/pods/fd78a1b4-e893-4254-8410-58f6313d7ea1/volumes"
Sep 16 20:21:36.391896 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.391864    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"]
Sep 16 20:21:36.392354 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.392226    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd78a1b4-e893-4254-8410-58f6313d7ea1" containerName="main"
Sep 16 20:21:36.392354 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.392241    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd78a1b4-e893-4254-8410-58f6313d7ea1" containerName="main"
Sep 16 20:21:36.392354 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.392316    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd78a1b4-e893-4254-8410-58f6313d7ea1" containerName="main"
Sep 16 20:21:36.405200 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.405171    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.405479 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.405455    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"]
Sep 16 20:21:36.407890 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.407864    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 16 20:21:36.416805 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.416780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:21:36.539133 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.539097    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-model-cache\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.539329 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.539308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tls-certs\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.539432 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.539364    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-home\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.539503 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.539483    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-dshm\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.539563 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.539511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7mf\" (UniqueName: \"kubernetes.io/projected/9d9aad89-bf01-42d5-a07d-70a121f8bc87-kube-api-access-8t7mf\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.539563 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.539554    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tmp-dir\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.640964 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.640924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-model-cache\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.641172 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.640973    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tls-certs\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.641172 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.641005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-home\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.641172 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.641055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-dshm\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.641172 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.641077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7mf\" (UniqueName: \"kubernetes.io/projected/9d9aad89-bf01-42d5-a07d-70a121f8bc87-kube-api-access-8t7mf\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.641172 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.641109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tmp-dir\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.641449 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.641382    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-model-cache\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.641449 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.641435    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-home\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.641573 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.641560    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tmp-dir\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.643293 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.643233    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-dshm\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.643532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.643516    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tls-certs\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.649753 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.649727    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7mf\" (UniqueName: \"kubernetes.io/projected/9d9aad89-bf01-42d5-a07d-70a121f8bc87-kube-api-access-8t7mf\") pod \"late-v2-kserve-86766546f-kdxps\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.715715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.715684    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:36.990122 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:36.990091    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"]
Sep 16 20:21:36.992116 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:21:36.992090    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9aad89_bf01_42d5_a07d_70a121f8bc87.slice/crio-94b88c1d0aa7881c1855ab153491f0dcd467c8dd8c56e8bafb75b66f471aa421 WatchSource:0}: Error finding container 94b88c1d0aa7881c1855ab153491f0dcd467c8dd8c56e8bafb75b66f471aa421: Status 404 returned error can't find the container with id 94b88c1d0aa7881c1855ab153491f0dcd467c8dd8c56e8bafb75b66f471aa421
Sep 16 20:21:37.968728 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:37.968687    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps" event={"ID":"9d9aad89-bf01-42d5-a07d-70a121f8bc87","Type":"ContainerStarted","Data":"8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6"}
Sep 16 20:21:37.969104 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:37.968751    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps" event={"ID":"9d9aad89-bf01-42d5-a07d-70a121f8bc87","Type":"ContainerStarted","Data":"94b88c1d0aa7881c1855ab153491f0dcd467c8dd8c56e8bafb75b66f471aa421"}
Sep 16 20:21:37.988705 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:37.988642    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps" podStartSLOduration=1.9886181120000002 podStartE2EDuration="1.988618112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:21:37.985335461 +0000 UTC m=+2529.542680558" watchObservedRunningTime="2026-09-16 20:21:37.988618112 +0000 UTC m=+2529.545963202"
Sep 16 20:21:38.716183 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:38.716144    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:46.716926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:46.716876    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:46.729715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:46.729693    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:21:47.028569 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:21:47.028493    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:22:19.439077 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:22:19.439033    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:22:19.520243 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:22:19.520212    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:22:26.389329 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:22:26.389290    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:22:46.545882 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:22:46.545839    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:22:46.548055 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:22:46.547158    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"]
Sep 16 20:22:46.640174 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:22:46.640133    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:22:46.785334 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:22:46.785303    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"]
Sep 16 20:22:46.800678 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:22:46.800591    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:22:51.562862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:22:51.562717    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:01.566733 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.566670    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps" podUID="9d9aad89-bf01-42d5-a07d-70a121f8bc87" containerName="main" containerID="cri-o://8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6" gracePeriod=45
Sep 16 20:23:01.825295 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.825229    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:23:01.841935 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.841829    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h" podUID="dad22229-e045-46d9-96dc-eda72f8e49df" containerName="main" containerID="cri-o://7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd" gracePeriod=45
Sep 16 20:23:01.908428 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908369    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-model-cache\") pod \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") "
Sep 16 20:23:01.908600 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908455    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-home\") pod \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") "
Sep 16 20:23:01.908600 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908491    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tls-certs\") pod \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") "
Sep 16 20:23:01.908600 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908513    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t7mf\" (UniqueName: \"kubernetes.io/projected/9d9aad89-bf01-42d5-a07d-70a121f8bc87-kube-api-access-8t7mf\") pod \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") "
Sep 16 20:23:01.908600 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908555    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tmp-dir\") pod \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") "
Sep 16 20:23:01.908600 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908576    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-dshm\") pod \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\" (UID: \"9d9aad89-bf01-42d5-a07d-70a121f8bc87\") "
Sep 16 20:23:01.908882 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908659    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d9aad89-bf01-42d5-a07d-70a121f8bc87" (UID: "9d9aad89-bf01-42d5-a07d-70a121f8bc87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:01.908882 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908743    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-home" (OuterVolumeSpecName: "home") pod "9d9aad89-bf01-42d5-a07d-70a121f8bc87" (UID: "9d9aad89-bf01-42d5-a07d-70a121f8bc87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:01.908882 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908842    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d9aad89-bf01-42d5-a07d-70a121f8bc87" (UID: "9d9aad89-bf01-42d5-a07d-70a121f8bc87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:01.908882 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908869    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:01.908882 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.908883    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:01.910800 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.910779    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-dshm" (OuterVolumeSpecName: "dshm") pod "9d9aad89-bf01-42d5-a07d-70a121f8bc87" (UID: "9d9aad89-bf01-42d5-a07d-70a121f8bc87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:01.910886 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.910827    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9aad89-bf01-42d5-a07d-70a121f8bc87-kube-api-access-8t7mf" (OuterVolumeSpecName: "kube-api-access-8t7mf") pod "9d9aad89-bf01-42d5-a07d-70a121f8bc87" (UID: "9d9aad89-bf01-42d5-a07d-70a121f8bc87"). InnerVolumeSpecName "kube-api-access-8t7mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:23:01.910886 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:01.910848    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d9aad89-bf01-42d5-a07d-70a121f8bc87" (UID: "9d9aad89-bf01-42d5-a07d-70a121f8bc87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:23:02.009971 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.009937    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.009971 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.009967    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7mf\" (UniqueName: \"kubernetes.io/projected/9d9aad89-bf01-42d5-a07d-70a121f8bc87-kube-api-access-8t7mf\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.009971 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.009976    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.010246 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.009985    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d9aad89-bf01-42d5-a07d-70a121f8bc87-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.076646 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.076566    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:23:02.111240 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111207    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-tmp-dir\") pod \"dad22229-e045-46d9-96dc-eda72f8e49df\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") "
Sep 16 20:23:02.111240 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111247    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f66xm\" (UniqueName: \"kubernetes.io/projected/dad22229-e045-46d9-96dc-eda72f8e49df-kube-api-access-f66xm\") pod \"dad22229-e045-46d9-96dc-eda72f8e49df\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") "
Sep 16 20:23:02.111509 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111289    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad22229-e045-46d9-96dc-eda72f8e49df-tls-certs\") pod \"dad22229-e045-46d9-96dc-eda72f8e49df\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") "
Sep 16 20:23:02.111509 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111307    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-dshm\") pod \"dad22229-e045-46d9-96dc-eda72f8e49df\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") "
Sep 16 20:23:02.111509 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111367    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-home\") pod \"dad22229-e045-46d9-96dc-eda72f8e49df\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") "
Sep 16 20:23:02.111509 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111419    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-model-cache\") pod \"dad22229-e045-46d9-96dc-eda72f8e49df\" (UID: \"dad22229-e045-46d9-96dc-eda72f8e49df\") "
Sep 16 20:23:02.111719 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111560    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dad22229-e045-46d9-96dc-eda72f8e49df" (UID: "dad22229-e045-46d9-96dc-eda72f8e49df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:02.111719 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111686    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-home" (OuterVolumeSpecName: "home") pod "dad22229-e045-46d9-96dc-eda72f8e49df" (UID: "dad22229-e045-46d9-96dc-eda72f8e49df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:02.111719 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111706    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.111867 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.111770    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-model-cache" (OuterVolumeSpecName: "model-cache") pod "dad22229-e045-46d9-96dc-eda72f8e49df" (UID: "dad22229-e045-46d9-96dc-eda72f8e49df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:02.113719 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.113697    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-dshm" (OuterVolumeSpecName: "dshm") pod "dad22229-e045-46d9-96dc-eda72f8e49df" (UID: "dad22229-e045-46d9-96dc-eda72f8e49df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:23:02.113966 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.113945    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad22229-e045-46d9-96dc-eda72f8e49df-kube-api-access-f66xm" (OuterVolumeSpecName: "kube-api-access-f66xm") pod "dad22229-e045-46d9-96dc-eda72f8e49df" (UID: "dad22229-e045-46d9-96dc-eda72f8e49df"). InnerVolumeSpecName "kube-api-access-f66xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:23:02.114014 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.113948    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad22229-e045-46d9-96dc-eda72f8e49df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dad22229-e045-46d9-96dc-eda72f8e49df" (UID: "dad22229-e045-46d9-96dc-eda72f8e49df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:23:02.212531 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.212487    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.212531 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.212523    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.212531 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.212539    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66xm\" (UniqueName: \"kubernetes.io/projected/dad22229-e045-46d9-96dc-eda72f8e49df-kube-api-access-f66xm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.212781 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.212551    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad22229-e045-46d9-96dc-eda72f8e49df-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.212781 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.212562    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad22229-e045-46d9-96dc-eda72f8e49df-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:23:02.407723 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.407687    2574 generic.go:356] "Generic (PLEG): container finished" podID="dad22229-e045-46d9-96dc-eda72f8e49df" containerID="7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd" exitCode=0
Sep 16 20:23:02.407932 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.407738    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"
Sep 16 20:23:02.407932 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.407768    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h" event={"ID":"dad22229-e045-46d9-96dc-eda72f8e49df","Type":"ContainerDied","Data":"7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd"}
Sep 16 20:23:02.407932 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.407813    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h" event={"ID":"dad22229-e045-46d9-96dc-eda72f8e49df","Type":"ContainerDied","Data":"c320e81105c9587056a9b9072c71af92547e23358a3cc5854ccb0a1b31ede9fb"}
Sep 16 20:23:02.407932 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.407835    2574 scope.go:117] "RemoveContainer" containerID="7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd"
Sep 16 20:23:02.409248 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.409221    2574 generic.go:356] "Generic (PLEG): container finished" podID="9d9aad89-bf01-42d5-a07d-70a121f8bc87" containerID="8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6" exitCode=0
Sep 16 20:23:02.409361 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.409258    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps" event={"ID":"9d9aad89-bf01-42d5-a07d-70a121f8bc87","Type":"ContainerDied","Data":"8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6"}
Sep 16 20:23:02.409361 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.409271    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"
Sep 16 20:23:02.409361 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.409293    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps" event={"ID":"9d9aad89-bf01-42d5-a07d-70a121f8bc87","Type":"ContainerDied","Data":"94b88c1d0aa7881c1855ab153491f0dcd467c8dd8c56e8bafb75b66f471aa421"}
Sep 16 20:23:02.417099 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.417082    2574 scope.go:117] "RemoveContainer" containerID="7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd"
Sep 16 20:23:02.417454 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:23:02.417431    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd\": container with ID starting with 7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd not found: ID does not exist" containerID="7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd"
Sep 16 20:23:02.417536 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.417463    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd"} err="failed to get container status \"7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd\": rpc error: code = NotFound desc = could not find container \"7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd\": container with ID starting with 7c994234adce4a586696abcc2769fa9e0f20048abd68cb0762152d358f5a78cd not found: ID does not exist"
Sep 16 20:23:02.417536 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.417486    2574 scope.go:117] "RemoveContainer" containerID="8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6"
Sep 16 20:23:02.425813 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.425761    2574 scope.go:117] "RemoveContainer" containerID="8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6"
Sep 16 20:23:02.426054 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:23:02.426036    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6\": container with ID starting with 8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6 not found: ID does not exist" containerID="8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6"
Sep 16 20:23:02.426104 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.426060    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6"} err="failed to get container status \"8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6\": rpc error: code = NotFound desc = could not find container \"8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6\": container with ID starting with 8522da1ed22f4551b809adf6a5f4212a95e5edf79b708a54c5c161d3d10e55b6 not found: ID does not exist"
Sep 16 20:23:02.434715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.434686    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"]
Sep 16 20:23:02.438020 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.437998    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-xzx5h"]
Sep 16 20:23:02.448766 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.448742    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"]
Sep 16 20:23:02.450530 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:02.450511    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-kdxps"]
Sep 16 20:23:03.088579 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:03.088545    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d9aad89-bf01-42d5-a07d-70a121f8bc87" path="/var/lib/kubelet/pods/9d9aad89-bf01-42d5-a07d-70a121f8bc87/volumes"
Sep 16 20:23:03.088951 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:03.088942    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad22229-e045-46d9-96dc-eda72f8e49df" path="/var/lib/kubelet/pods/dad22229-e045-46d9-96dc-eda72f8e49df/volumes"
Sep 16 20:23:08.523789 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:08.523750    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:08.622136 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:08.622102    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:15.476105 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:15.476067    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:21.428577 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:21.428533    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:21.529384 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:21.529354    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:23.031044 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:23.031004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:28.668606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:28.668574    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:34.828483 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:34.828437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:34.930738 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:34.930699    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:23:38.784483 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:23:38.784442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:24:01.390549 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.390516    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"]
Sep 16 20:24:01.391033 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.390873    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d9aad89-bf01-42d5-a07d-70a121f8bc87" containerName="main"
Sep 16 20:24:01.391033 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.390885    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d9aad89-bf01-42d5-a07d-70a121f8bc87" containerName="main"
Sep 16 20:24:01.391033 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.390895    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad22229-e045-46d9-96dc-eda72f8e49df" containerName="main"
Sep 16 20:24:01.391033 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.390900    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad22229-e045-46d9-96dc-eda72f8e49df" containerName="main"
Sep 16 20:24:01.391033 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.390979    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d9aad89-bf01-42d5-a07d-70a121f8bc87" containerName="main"
Sep 16 20:24:01.391033 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.390986    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad22229-e045-46d9-96dc-eda72f8e49df" containerName="main"
Sep 16 20:24:01.394061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.394040    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.396855 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.396833    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 20:24:01.398006 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.397983    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 20:24:01.398138 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.398025    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 20:24:01.398138 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.398076    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-gcz2m\""
Sep 16 20:24:01.403552 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.403527    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"]
Sep 16 20:24:01.417921 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.417892    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.418061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.417940    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ct7\" (UniqueName: \"kubernetes.io/projected/80727c91-e6c1-4c6e-b50d-7ba14bded604-kube-api-access-45ct7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.418061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.417972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.418061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.418036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.418196 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.418085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80727c91-e6c1-4c6e-b50d-7ba14bded604-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.418196 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.418122    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.422254 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.422225    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:24:01.519552 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.519514    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80727c91-e6c1-4c6e-b50d-7ba14bded604-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.519552 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.519552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.519826 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.519654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.519826 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.519704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ct7\" (UniqueName: \"kubernetes.io/projected/80727c91-e6c1-4c6e-b50d-7ba14bded604-kube-api-access-45ct7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.519826 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.519729    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.519826 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.519772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.520053 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.519907    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.520053 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.520000    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.520150 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.520109    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.521972 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.521943    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.522121 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.522105    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80727c91-e6c1-4c6e-b50d-7ba14bded604-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.528137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.528117    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ct7\" (UniqueName: \"kubernetes.io/projected/80727c91-e6c1-4c6e-b50d-7ba14bded604-kube-api-access-45ct7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.706595 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.706517    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:01.983673 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:01.983645    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"]
Sep 16 20:24:01.986179 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:24:01.986155    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80727c91_e6c1_4c6e_b50d_7ba14bded604.slice/crio-8e93ca481a4bf91d8fd46df3f0cf987410f2ef0ff4b2e4687ec58a66edc3fd54 WatchSource:0}: Error finding container 8e93ca481a4bf91d8fd46df3f0cf987410f2ef0ff4b2e4687ec58a66edc3fd54: Status 404 returned error can't find the container with id 8e93ca481a4bf91d8fd46df3f0cf987410f2ef0ff4b2e4687ec58a66edc3fd54
Sep 16 20:24:02.720692 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:02.720656    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t" event={"ID":"80727c91-e6c1-4c6e-b50d-7ba14bded604","Type":"ContainerStarted","Data":"50753aea9b0b897ebbea44c068912442e066d983a0c5c071428211f1c563564a"}
Sep 16 20:24:02.721095 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:02.720719    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t" event={"ID":"80727c91-e6c1-4c6e-b50d-7ba14bded604","Type":"ContainerStarted","Data":"8e93ca481a4bf91d8fd46df3f0cf987410f2ef0ff4b2e4687ec58a66edc3fd54"}
Sep 16 20:24:02.741275 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:02.741228    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t" podStartSLOduration=1.741210464 podStartE2EDuration="1.741210464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:24:02.73926726 +0000 UTC m=+2674.296612369" watchObservedRunningTime="2026-09-16 20:24:02.741210464 +0000 UTC m=+2674.298555551"
Sep 16 20:24:02.845954 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:02.845911    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:24:03.707198 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:03.707159    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:11.706790 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:11.706753    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:11.718934 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:11.718907    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:11.779017 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:11.778980    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:24:24.421902 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:24.421815    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:24:29.124695 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:29.124666    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:24:29.127279 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:29.127258    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:24:37.921099 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:37.921056    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:24:38.145721 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:38.145687    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:24:50.052800 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:50.052755    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:24:54.117121 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:24:54.117086    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:09.601236 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:09.601182    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:09.725902 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:09.725858    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:09.812313 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:09.812271    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:14.675844 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:14.675804    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:18.903569 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:18.903532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:36.912172 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.912137    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"]
Sep 16 20:25:36.915905 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.915884    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:36.920302 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.919458    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 20:25:36.921112 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.921074    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-89679\""
Sep 16 20:25:36.921745 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.921300    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 16 20:25:36.921745 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.921540    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 20:25:36.933984 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.933956    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"]
Sep 16 20:25:36.951298 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.951267    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:36.954319 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.954300    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzmd6\" (UniqueName: \"kubernetes.io/projected/c75fc0db-c1a8-408a-83cf-5b6a078eec02-kube-api-access-vzmd6\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:36.954383 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.954330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:36.954434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.954391    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:36.954472 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.954440    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:36.954512 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.954487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-home\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:36.954557 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:36.954542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-dshm\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.055035 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.054987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.055035 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.055036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.055277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.055093    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-home\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.055277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.055140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-dshm\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.055277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.055183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzmd6\" (UniqueName: \"kubernetes.io/projected/c75fc0db-c1a8-408a-83cf-5b6a078eec02-kube-api-access-vzmd6\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.055277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.055263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.055543 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.055520    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.055614 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.055549    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-home\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.055663 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.055640    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.057484 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.057453    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-dshm\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.057762 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.057743    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.064048 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.064017    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzmd6\" (UniqueName: \"kubernetes.io/projected/c75fc0db-c1a8-408a-83cf-5b6a078eec02-kube-api-access-vzmd6\") pod \"roll-v2-kserve-d4dbf9d67-j4kf6\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.228577 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.228495    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:37.502622 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:37.502598    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"]
Sep 16 20:25:37.504636 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:25:37.504601    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75fc0db_c1a8_408a_83cf_5b6a078eec02.slice/crio-ed2dd1bafa6b7880e0a0936346f945565944d3a9455b16f342e300c91c02a60d WatchSource:0}: Error finding container ed2dd1bafa6b7880e0a0936346f945565944d3a9455b16f342e300c91c02a60d: Status 404 returned error can't find the container with id ed2dd1bafa6b7880e0a0936346f945565944d3a9455b16f342e300c91c02a60d
Sep 16 20:25:38.236566 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:38.236528    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:38.239157 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:38.239118    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6" event={"ID":"c75fc0db-c1a8-408a-83cf-5b6a078eec02","Type":"ContainerStarted","Data":"e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4"}
Sep 16 20:25:38.239300 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:38.239213    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6" event={"ID":"c75fc0db-c1a8-408a-83cf-5b6a078eec02","Type":"ContainerStarted","Data":"ed2dd1bafa6b7880e0a0936346f945565944d3a9455b16f342e300c91c02a60d"}
Sep 16 20:25:38.269343 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:38.269266    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6" podStartSLOduration=2.269244056 podStartE2EDuration="2.269244056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:25:38.26458431 +0000 UTC m=+2769.821929417" watchObservedRunningTime="2026-09-16 20:25:38.269244056 +0000 UTC m=+2769.826589143"
Sep 16 20:25:38.298233 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:38.298183    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:39.229006 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:39.228970    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:47.229455 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:47.229416    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:47.243132 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:47.243107    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:47.296281 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:47.296250    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:25:51.949236 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:51.949154    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:57.792253 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:57.792199    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:25:57.880618 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:25:57.880578    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:28.822881 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:28.822830    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:28.827530 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:28.827503    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"]
Sep 16 20:26:28.836792 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:26:28.836756    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:26:28.836792 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:26:28.836792 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:26:28.836792 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="50753aea9b0b897ebbea44c068912442e066d983a0c5c071428211f1c563564a" cmd=["/bin/sleep","15"]
Sep 16 20:26:28.837111 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:28.836810    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t" message="" err=<
Sep 16 20:26:28.837111 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:26:28.837111 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:26:28.837111 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:26:28.837111 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:26:28.836825    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:26:28.837111 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:26:28.837111 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:26:28.837111 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t" podUID="80727c91-e6c1-4c6e-b50d-7ba14bded604" containerName="main" containerID="cri-o://50753aea9b0b897ebbea44c068912442e066d983a0c5c071428211f1c563564a"
Sep 16 20:26:28.837111 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:28.836875    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t" podUID="80727c91-e6c1-4c6e-b50d-7ba14bded604" containerName="main" containerID="cri-o://50753aea9b0b897ebbea44c068912442e066d983a0c5c071428211f1c563564a" gracePeriod=60
Sep 16 20:26:28.917672 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:28.917635    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:29.510738 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.510705    2574 generic.go:356] "Generic (PLEG): container finished" podID="80727c91-e6c1-4c6e-b50d-7ba14bded604" containerID="50753aea9b0b897ebbea44c068912442e066d983a0c5c071428211f1c563564a" exitCode=0
Sep 16 20:26:29.510921 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.510754    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t" event={"ID":"80727c91-e6c1-4c6e-b50d-7ba14bded604","Type":"ContainerDied","Data":"50753aea9b0b897ebbea44c068912442e066d983a0c5c071428211f1c563564a"}
Sep 16 20:26:29.587168 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.587140    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:26:29.635616 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.635575    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-model-cache\") pod \"80727c91-e6c1-4c6e-b50d-7ba14bded604\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") "
Sep 16 20:26:29.635785 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.635624    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-dshm\") pod \"80727c91-e6c1-4c6e-b50d-7ba14bded604\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") "
Sep 16 20:26:29.635785 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.635690    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45ct7\" (UniqueName: \"kubernetes.io/projected/80727c91-e6c1-4c6e-b50d-7ba14bded604-kube-api-access-45ct7\") pod \"80727c91-e6c1-4c6e-b50d-7ba14bded604\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") "
Sep 16 20:26:29.635785 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.635733    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80727c91-e6c1-4c6e-b50d-7ba14bded604-tls-certs\") pod \"80727c91-e6c1-4c6e-b50d-7ba14bded604\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") "
Sep 16 20:26:29.635785 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.635760    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-home\") pod \"80727c91-e6c1-4c6e-b50d-7ba14bded604\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") "
Sep 16 20:26:29.636010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.635831    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-model-cache" (OuterVolumeSpecName: "model-cache") pod "80727c91-e6c1-4c6e-b50d-7ba14bded604" (UID: "80727c91-e6c1-4c6e-b50d-7ba14bded604"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:29.636010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.635858    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-tmp-dir\") pod \"80727c91-e6c1-4c6e-b50d-7ba14bded604\" (UID: \"80727c91-e6c1-4c6e-b50d-7ba14bded604\") "
Sep 16 20:26:29.636126 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.636044    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-home" (OuterVolumeSpecName: "home") pod "80727c91-e6c1-4c6e-b50d-7ba14bded604" (UID: "80727c91-e6c1-4c6e-b50d-7ba14bded604"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:29.636203 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.636129    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80727c91-e6c1-4c6e-b50d-7ba14bded604" (UID: "80727c91-e6c1-4c6e-b50d-7ba14bded604"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:29.636203 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.636157    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:29.636203 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.636173    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:29.638703 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.638675    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-dshm" (OuterVolumeSpecName: "dshm") pod "80727c91-e6c1-4c6e-b50d-7ba14bded604" (UID: "80727c91-e6c1-4c6e-b50d-7ba14bded604"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:29.638820 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.638675    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80727c91-e6c1-4c6e-b50d-7ba14bded604-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80727c91-e6c1-4c6e-b50d-7ba14bded604" (UID: "80727c91-e6c1-4c6e-b50d-7ba14bded604"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:26:29.638820 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.638726    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80727c91-e6c1-4c6e-b50d-7ba14bded604-kube-api-access-45ct7" (OuterVolumeSpecName: "kube-api-access-45ct7") pod "80727c91-e6c1-4c6e-b50d-7ba14bded604" (UID: "80727c91-e6c1-4c6e-b50d-7ba14bded604"). InnerVolumeSpecName "kube-api-access-45ct7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:26:29.736978 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.736897    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:29.736978 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.736924    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ct7\" (UniqueName: \"kubernetes.io/projected/80727c91-e6c1-4c6e-b50d-7ba14bded604-kube-api-access-45ct7\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:29.736978 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.736934    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80727c91-e6c1-4c6e-b50d-7ba14bded604-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:29.736978 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:29.736944    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80727c91-e6c1-4c6e-b50d-7ba14bded604-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:30.517280 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:30.517244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t" event={"ID":"80727c91-e6c1-4c6e-b50d-7ba14bded604","Type":"ContainerDied","Data":"8e93ca481a4bf91d8fd46df3f0cf987410f2ef0ff4b2e4687ec58a66edc3fd54"}
Sep 16 20:26:30.517739 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:30.517299    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"
Sep 16 20:26:30.517739 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:30.517344    2574 scope.go:117] "RemoveContainer" containerID="50753aea9b0b897ebbea44c068912442e066d983a0c5c071428211f1c563564a"
Sep 16 20:26:30.539001 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:30.538977    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"]
Sep 16 20:26:30.542936 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:30.542912    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wsf4t"]
Sep 16 20:26:31.088884 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:31.088850    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80727c91-e6c1-4c6e-b50d-7ba14bded604" path="/var/lib/kubelet/pods/80727c91-e6c1-4c6e-b50d-7ba14bded604/volumes"
Sep 16 20:26:43.623290 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:43.623246    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:43.734541 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:43.734507    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"]
Sep 16 20:26:43.751153 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:43.751120    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:43.848531 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:43.848496    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:44.283092 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:44.283043    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:48.696218 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:48.696179    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:57.496117 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:57.496084    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:57.597147 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:57.597105    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:58.755166 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:58.755128    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6" podUID="c75fc0db-c1a8-408a-83cf-5b6a078eec02" containerName="main" containerID="cri-o://e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4" gracePeriod=45
Sep 16 20:26:59.009999 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.009943    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:26:59.101464 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.101435    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzmd6\" (UniqueName: \"kubernetes.io/projected/c75fc0db-c1a8-408a-83cf-5b6a078eec02-kube-api-access-vzmd6\") pod \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") "
Sep 16 20:26:59.101627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.101469    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tls-certs\") pod \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") "
Sep 16 20:26:59.101627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.101531    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-model-cache\") pod \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") "
Sep 16 20:26:59.101627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.101546    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-dshm\") pod \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") "
Sep 16 20:26:59.101627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.101589    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tmp-dir\") pod \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") "
Sep 16 20:26:59.101627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.101623    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-home\") pod \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\" (UID: \"c75fc0db-c1a8-408a-83cf-5b6a078eec02\") "
Sep 16 20:26:59.101869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.101806    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-model-cache" (OuterVolumeSpecName: "model-cache") pod "c75fc0db-c1a8-408a-83cf-5b6a078eec02" (UID: "c75fc0db-c1a8-408a-83cf-5b6a078eec02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:59.101933 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.101883    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c75fc0db-c1a8-408a-83cf-5b6a078eec02" (UID: "c75fc0db-c1a8-408a-83cf-5b6a078eec02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:59.102034 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.102009    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-home" (OuterVolumeSpecName: "home") pod "c75fc0db-c1a8-408a-83cf-5b6a078eec02" (UID: "c75fc0db-c1a8-408a-83cf-5b6a078eec02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:59.103778 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.103746    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75fc0db-c1a8-408a-83cf-5b6a078eec02-kube-api-access-vzmd6" (OuterVolumeSpecName: "kube-api-access-vzmd6") pod "c75fc0db-c1a8-408a-83cf-5b6a078eec02" (UID: "c75fc0db-c1a8-408a-83cf-5b6a078eec02"). InnerVolumeSpecName "kube-api-access-vzmd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:26:59.103878 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.103795    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c75fc0db-c1a8-408a-83cf-5b6a078eec02" (UID: "c75fc0db-c1a8-408a-83cf-5b6a078eec02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:26:59.103878 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.103798    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-dshm" (OuterVolumeSpecName: "dshm") pod "c75fc0db-c1a8-408a-83cf-5b6a078eec02" (UID: "c75fc0db-c1a8-408a-83cf-5b6a078eec02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:26:59.202528 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.202491    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.202528 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.202519    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzmd6\" (UniqueName: \"kubernetes.io/projected/c75fc0db-c1a8-408a-83cf-5b6a078eec02-kube-api-access-vzmd6\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.202528 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.202532    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.202826 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.202545    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.202826 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.202557    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.202826 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.202568    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c75fc0db-c1a8-408a-83cf-5b6a078eec02-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:26:59.265846 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.265758    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:59.331895 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.331854    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:26:59.677134 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.677101    2574 generic.go:356] "Generic (PLEG): container finished" podID="c75fc0db-c1a8-408a-83cf-5b6a078eec02" containerID="e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4" exitCode=0
Sep 16 20:26:59.677305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.677151    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"
Sep 16 20:26:59.677305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.677160    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6" event={"ID":"c75fc0db-c1a8-408a-83cf-5b6a078eec02","Type":"ContainerDied","Data":"e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4"}
Sep 16 20:26:59.677305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.677199    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6" event={"ID":"c75fc0db-c1a8-408a-83cf-5b6a078eec02","Type":"ContainerDied","Data":"ed2dd1bafa6b7880e0a0936346f945565944d3a9455b16f342e300c91c02a60d"}
Sep 16 20:26:59.677305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.677219    2574 scope.go:117] "RemoveContainer" containerID="e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4"
Sep 16 20:26:59.686586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.686568    2574 scope.go:117] "RemoveContainer" containerID="e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4"
Sep 16 20:26:59.686850 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:26:59.686831    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4\": container with ID starting with e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4 not found: ID does not exist" containerID="e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4"
Sep 16 20:26:59.686900 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.686860    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4"} err="failed to get container status \"e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4\": rpc error: code = NotFound desc = could not find container \"e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4\": container with ID starting with e90c832389d21b731f7bb0dda01cf53da88d528295600b939406972957af0ed4 not found: ID does not exist"
Sep 16 20:26:59.703665 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.703636    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"]
Sep 16 20:26:59.707230 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:26:59.707206    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-j4kf6"]
Sep 16 20:27:01.088568 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:01.088536    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c75fc0db-c1a8-408a-83cf-5b6a078eec02" path="/var/lib/kubelet/pods/c75fc0db-c1a8-408a-83cf-5b6a078eec02/volumes"
Sep 16 20:27:07.016653 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:07.016617    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:12.236704 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:12.236671    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:14.472222 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:14.472183    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:18.077034 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:18.077000    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:18.182458 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:18.182427    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:18.459851 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:18.459820    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:18.560531 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:18.560494    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:24.977329 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:24.977241    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:30.955191 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:30.955154    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:31.048173 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:31.048133    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:40.621711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:40.621670    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:40.890875 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:40.890788    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:57.028542 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:57.028500    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:27:58.423763 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:27:58.423726    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:06.280322 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:06.280287    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:06.339304 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:06.339270    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:20.687286 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.687222    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:20.788054 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.788007    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:20.911274 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.911235    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:20.963062 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.963027    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"]
Sep 16 20:28:20.963640 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.963619    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c75fc0db-c1a8-408a-83cf-5b6a078eec02" containerName="main"
Sep 16 20:28:20.963706 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.963644    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c75fc0db-c1a8-408a-83cf-5b6a078eec02" containerName="main"
Sep 16 20:28:20.963706 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.963681    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80727c91-e6c1-4c6e-b50d-7ba14bded604" containerName="main"
Sep 16 20:28:20.963706 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.963690    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="80727c91-e6c1-4c6e-b50d-7ba14bded604" containerName="main"
Sep 16 20:28:20.963803 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.963786    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="80727c91-e6c1-4c6e-b50d-7ba14bded604" containerName="main"
Sep 16 20:28:20.963803 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.963796    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c75fc0db-c1a8-408a-83cf-5b6a078eec02" containerName="main"
Sep 16 20:28:20.967293 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.967271    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:20.970625 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.970603    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-bxz8j\""
Sep 16 20:28:20.971848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.971832    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 20:28:20.972961 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.972943    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 16 20:28:20.973058 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.972983    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 20:28:20.984270 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:20.984240    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"]
Sep 16 20:28:21.002734 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.002698    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-socket\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.002926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.002743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-data\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.002926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.002796    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-envoy\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.002926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.002848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kb2\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-kube-api-access-25kb2\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.002926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.002878    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.003164 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.002935    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-podinfo\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.003164 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.002978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-token\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.003164 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.003097    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-credential-socket\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.003495 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.003175    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-certs\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.009290 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.009247    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:21.102931 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.102893    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:21.103848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.103830    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-token\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.103953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.103868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-credential-socket\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.103953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.103890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-certs\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.103953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.103911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-socket\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.103953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.103930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-data\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.103953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.103950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-envoy\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.104219 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.103984    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kb2\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-kube-api-access-25kb2\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.104219 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.104005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.104219 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.104035    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-podinfo\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.104480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.104302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-credential-socket\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.104480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.104336    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-socket\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.104480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.104382    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-data\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.104654 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.104551    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-certs\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.104954 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.104931    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.106380 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.106360    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-envoy\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.106624 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.106608    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-podinfo\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.112387 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.112362    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-token\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.112579 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.112539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kb2\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-kube-api-access-25kb2\") pod \"router-gateway-1-openshift-default-699c764795-sgmvw\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.279028 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.278935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:21.578572 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.578543    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"]
Sep 16 20:28:21.581951 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:28:21.581917    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ea18d8_8a9b_4062_ad7e_f2aefee63190.slice/crio-7e9aa20dd1a7fc9a22e4da1b55d64e7a855321f2edff1ed262eaec2ed1f62ad2 WatchSource:0}: Error finding container 7e9aa20dd1a7fc9a22e4da1b55d64e7a855321f2edff1ed262eaec2ed1f62ad2: Status 404 returned error can't find the container with id 7e9aa20dd1a7fc9a22e4da1b55d64e7a855321f2edff1ed262eaec2ed1f62ad2
Sep 16 20:28:21.584721 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.584683    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:28:21.584841 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.584766    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:28:21.584841 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:21.584802    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:28:22.098611 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:22.098575    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" event={"ID":"b2ea18d8-8a9b-4062-ad7e-f2aefee63190","Type":"ContainerStarted","Data":"f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5"}
Sep 16 20:28:22.099012 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:22.098635    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" event={"ID":"b2ea18d8-8a9b-4062-ad7e-f2aefee63190","Type":"ContainerStarted","Data":"7e9aa20dd1a7fc9a22e4da1b55d64e7a855321f2edff1ed262eaec2ed1f62ad2"}
Sep 16 20:28:22.130944 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:22.130896    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" podStartSLOduration=2.130881265 podStartE2EDuration="2.130881265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:28:22.128555384 +0000 UTC m=+2933.685900500" watchObservedRunningTime="2026-09-16 20:28:22.130881265 +0000 UTC m=+2933.688226351"
Sep 16 20:28:22.279935 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:22.279901    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:22.285174 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:22.285146    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:23.108424 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:23.105523    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:23.111188 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:23.111160    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:24.444384 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:24.444345    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:25.765314 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:25.765278    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:27.886994 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:27.886959    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:32.391087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.391057    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"]
Sep 16 20:28:32.394837 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.394816    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.399276 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.398817    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 16 20:28:32.399276 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.399054    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-8k5h8\""
Sep 16 20:28:32.403289 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.403262    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"]
Sep 16 20:28:32.408711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.408687    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:28:32.505555 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.505523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.505555 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.505559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh2xt\" (UniqueName: \"kubernetes.io/projected/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-kube-api-access-kh2xt\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.505778 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.505584    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.505778 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.505614    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.505778 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.505718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.505877 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.505775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.606268 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.606231    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.606472 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.606292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.606472 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.606424    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh2xt\" (UniqueName: \"kubernetes.io/projected/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-kube-api-access-kh2xt\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.606571 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.606477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.606571 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.606503    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.606675 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.606583    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.606836 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.606817    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.606943 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.606923    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.607011 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.606979    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.608765 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.608750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.608876 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.608857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.614918 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.614896    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh2xt\" (UniqueName: \"kubernetes.io/projected/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-kube-api-access-kh2xt\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.708179 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.708074    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:32.982996 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:32.982960    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"]
Sep 16 20:28:32.984352 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:28:32.984309    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bbf6b5_3531_47c6_b6c8_59dcaf30b94b.slice/crio-b7a64b7d222db3b50c5941f0826adb60e058b1003201ca14f0071c67d4ba007d WatchSource:0}: Error finding container b7a64b7d222db3b50c5941f0826adb60e058b1003201ca14f0071c67d4ba007d: Status 404 returned error can't find the container with id b7a64b7d222db3b50c5941f0826adb60e058b1003201ca14f0071c67d4ba007d
Sep 16 20:28:33.158546 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:33.158500    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl" event={"ID":"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b","Type":"ContainerStarted","Data":"85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f"}
Sep 16 20:28:33.158758 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:33.158588    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl" event={"ID":"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b","Type":"ContainerStarted","Data":"b7a64b7d222db3b50c5941f0826adb60e058b1003201ca14f0071c67d4ba007d"}
Sep 16 20:28:33.190876 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:33.190798    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl" podStartSLOduration=1.190779175 podStartE2EDuration="1.190779175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:28:33.189307545 +0000 UTC m=+2944.746652643" watchObservedRunningTime="2026-09-16 20:28:33.190779175 +0000 UTC m=+2944.748124261"
Sep 16 20:28:33.708507 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:33.708467    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:42.214477 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:42.214442    2574 generic.go:356] "Generic (PLEG): container finished" podID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerID="f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5" exitCode=0
Sep 16 20:28:42.215009 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:42.214520    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" event={"ID":"b2ea18d8-8a9b-4062-ad7e-f2aefee63190","Type":"ContainerDied","Data":"f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5"}
Sep 16 20:28:42.215009 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:42.215003    2574 scope.go:117] "RemoveContainer" containerID="f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5"
Sep 16 20:28:42.216026 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:42.215997    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:28:42.216119 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:42.216065    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:28:42.216119 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:42.216096    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:28:42.708823 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:42.708775    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:42.727944 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:42.727912    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:43.221587 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:43.221554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" event={"ID":"b2ea18d8-8a9b-4062-ad7e-f2aefee63190","Type":"ContainerStarted","Data":"deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242"}
Sep 16 20:28:43.233060 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:43.233032    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:28:43.279729 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:43.279692    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:43.279942 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:43.279912    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 20:28:44.280266 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:44.280218    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 20:28:45.279636 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:45.279591    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 20:28:46.279887 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:46.279840    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 20:28:47.279482 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:47.279434    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 20:28:48.280164 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:48.280122    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 20:28:49.283992 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:49.283959    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:49.284439 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:49.284238    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:28:49.284974 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:28:49.284958    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:29:03.065693 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:03.065607    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:29:09.321751 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:09.321712    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:29:09.375959 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:09.375923    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:29:26.501695 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:26.501644    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:29:26.575715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:26.575679    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:29:29.160913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:29.160869    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:29:29.163985 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:29.163958    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:29:44.916631 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:44.916592    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"]
Sep 16 20:29:44.925380 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:29:44.925342    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:29:44.925380 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:29:44.925380 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:29:44.925380 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f" cmd=["/bin/sleep","15"]
Sep 16 20:29:44.925667 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:44.925396    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl" message="" err=<
Sep 16 20:29:44.925667 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:29:44.925667 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:29:44.925667 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:29:44.925667 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:29:44.925434    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:29:44.925667 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:29:44.925667 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:29:44.925667 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl" podUID="04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" containerName="main" containerID="cri-o://85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f"
Sep 16 20:29:44.925667 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:44.925479    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl" podUID="04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" containerName="main" containerID="cri-o://85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f" gracePeriod=60
Sep 16 20:29:44.932022 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:44.931995    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:29:45.183579 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.183555    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:29:45.357099 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357060    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tls-certs\") pod \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") "
Sep 16 20:29:45.357298 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357112    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh2xt\" (UniqueName: \"kubernetes.io/projected/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-kube-api-access-kh2xt\") pod \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") "
Sep 16 20:29:45.357298 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357147    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-dshm\") pod \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") "
Sep 16 20:29:45.357298 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357169    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tmp-dir\") pod \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") "
Sep 16 20:29:45.357298 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357278    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-home\") pod \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") "
Sep 16 20:29:45.357575 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357316    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-model-cache\") pod \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\" (UID: \"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b\") "
Sep 16 20:29:45.357575 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357435    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" (UID: "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:29:45.357697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357545    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-home" (OuterVolumeSpecName: "home") pod "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" (UID: "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:29:45.357697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357617    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-model-cache" (OuterVolumeSpecName: "model-cache") pod "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" (UID: "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:29:45.357697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357685    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:29:45.357858 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.357702    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:29:45.359582 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.359556    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-kube-api-access-kh2xt" (OuterVolumeSpecName: "kube-api-access-kh2xt") pod "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" (UID: "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b"). InnerVolumeSpecName "kube-api-access-kh2xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:29:45.359696 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.359559    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" (UID: "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:29:45.359696 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.359590    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-dshm" (OuterVolumeSpecName: "dshm") pod "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" (UID: "04bbf6b5-3531-47c6-b6c8-59dcaf30b94b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:29:45.458518 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.458384    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh2xt\" (UniqueName: \"kubernetes.io/projected/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-kube-api-access-kh2xt\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:29:45.458518 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.458454    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:29:45.458518 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.458470    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:29:45.458518 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.458485    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:29:45.556980 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.556947    2574 generic.go:356] "Generic (PLEG): container finished" podID="04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" containerID="85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f" exitCode=0
Sep 16 20:29:45.557151 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.556995    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"
Sep 16 20:29:45.557151 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.557035    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl" event={"ID":"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b","Type":"ContainerDied","Data":"85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f"}
Sep 16 20:29:45.557151 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.557094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl" event={"ID":"04bbf6b5-3531-47c6-b6c8-59dcaf30b94b","Type":"ContainerDied","Data":"b7a64b7d222db3b50c5941f0826adb60e058b1003201ca14f0071c67d4ba007d"}
Sep 16 20:29:45.557151 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.557124    2574 scope.go:117] "RemoveContainer" containerID="85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f"
Sep 16 20:29:45.567227 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.567206    2574 scope.go:117] "RemoveContainer" containerID="85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f"
Sep 16 20:29:45.567567 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:29:45.567548    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f\": container with ID starting with 85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f not found: ID does not exist" containerID="85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f"
Sep 16 20:29:45.567631 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.567578    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f"} err="failed to get container status \"85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f\": rpc error: code = NotFound desc = could not find container \"85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f\": container with ID starting with 85820f6e9303cef8a4accd6b4e5e2bb51289128b21fe389ff968b4a35d9e838f not found: ID does not exist"
Sep 16 20:29:45.580106 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.580074    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"]
Sep 16 20:29:45.583747 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:45.583719    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qk6xl"]
Sep 16 20:29:47.088072 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:47.088032    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" path="/var/lib/kubelet/pods/04bbf6b5-3531-47c6-b6c8-59dcaf30b94b/volumes"
Sep 16 20:29:49.947199 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:49.947164    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:29:50.043414 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:50.043363    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:29:54.895294 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:54.895254    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"]
Sep 16 20:29:54.895761 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:54.895501    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy" containerID="cri-o://deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242" gracePeriod=30
Sep 16 20:29:54.989545 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:29:54.989511    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:00.142231 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.142201    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:30:00.177457 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177335    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istiod-ca-cert\") pod \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") "
Sep 16 20:30:00.177457 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177441    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-envoy\") pod \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") "
Sep 16 20:30:00.177715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177464    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-token\") pod \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") "
Sep 16 20:30:00.177715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177488    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25kb2\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-kube-api-access-25kb2\") pod \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") "
Sep 16 20:30:00.177715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-podinfo\") pod \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") "
Sep 16 20:30:00.177715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177553    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-socket\") pod \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") "
Sep 16 20:30:00.177715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177573    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-data\") pod \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") "
Sep 16 20:30:00.177715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177597    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-credential-socket\") pod \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") "
Sep 16 20:30:00.177715 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177634    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-certs\") pod \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\" (UID: \"b2ea18d8-8a9b-4062-ad7e-f2aefee63190\") "
Sep 16 20:30:00.178100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177815    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2ea18d8-8a9b-4062-ad7e-f2aefee63190" (UID: "b2ea18d8-8a9b-4062-ad7e-f2aefee63190"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:30:00.178100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177839    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2ea18d8-8a9b-4062-ad7e-f2aefee63190" (UID: "b2ea18d8-8a9b-4062-ad7e-f2aefee63190"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:30:00.178100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.177877    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2ea18d8-8a9b-4062-ad7e-f2aefee63190" (UID: "b2ea18d8-8a9b-4062-ad7e-f2aefee63190"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:30:00.178100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.178018    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2ea18d8-8a9b-4062-ad7e-f2aefee63190" (UID: "b2ea18d8-8a9b-4062-ad7e-f2aefee63190"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:30:00.178100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.178077    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2ea18d8-8a9b-4062-ad7e-f2aefee63190" (UID: "b2ea18d8-8a9b-4062-ad7e-f2aefee63190"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:30:00.178711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.178618    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:30:00.178711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.178642    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-data\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:30:00.178711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.178658    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-credential-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:30:00.178711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.178673    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-workload-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:30:00.178711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.178687    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istiod-ca-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:30:00.180310 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.180282    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2ea18d8-8a9b-4062-ad7e-f2aefee63190" (UID: "b2ea18d8-8a9b-4062-ad7e-f2aefee63190"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:30:00.180478 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.180285    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2ea18d8-8a9b-4062-ad7e-f2aefee63190" (UID: "b2ea18d8-8a9b-4062-ad7e-f2aefee63190"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:30:00.180478 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.180282    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-kube-api-access-25kb2" (OuterVolumeSpecName: "kube-api-access-25kb2") pod "b2ea18d8-8a9b-4062-ad7e-f2aefee63190" (UID: "b2ea18d8-8a9b-4062-ad7e-f2aefee63190"). InnerVolumeSpecName "kube-api-access-25kb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:30:00.180608 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.180475    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2ea18d8-8a9b-4062-ad7e-f2aefee63190" (UID: "b2ea18d8-8a9b-4062-ad7e-f2aefee63190"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 20:30:00.191186 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.191150    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:00.279387 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.279360    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-envoy\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:30:00.279387 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.279383    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-token\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:30:00.279387 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.279393    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25kb2\" (UniqueName: \"kubernetes.io/projected/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-kube-api-access-25kb2\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:30:00.279625 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.279418    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2ea18d8-8a9b-4062-ad7e-f2aefee63190-istio-podinfo\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:30:00.638862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.638827    2574 generic.go:356] "Generic (PLEG): container finished" podID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerID="deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242" exitCode=0
Sep 16 20:30:00.639087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.638905    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"
Sep 16 20:30:00.639087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.638912    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" event={"ID":"b2ea18d8-8a9b-4062-ad7e-f2aefee63190","Type":"ContainerDied","Data":"deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242"}
Sep 16 20:30:00.639087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.638965    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw" event={"ID":"b2ea18d8-8a9b-4062-ad7e-f2aefee63190","Type":"ContainerDied","Data":"7e9aa20dd1a7fc9a22e4da1b55d64e7a855321f2edff1ed262eaec2ed1f62ad2"}
Sep 16 20:30:00.639087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.638989    2574 scope.go:117] "RemoveContainer" containerID="deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242"
Sep 16 20:30:00.649951 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.649932    2574 scope.go:117] "RemoveContainer" containerID="f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5"
Sep 16 20:30:00.658171 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.658150    2574 scope.go:117] "RemoveContainer" containerID="deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242"
Sep 16 20:30:00.658486 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:30:00.658467    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242\": container with ID starting with deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242 not found: ID does not exist" containerID="deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242"
Sep 16 20:30:00.658578 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.658501    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242"} err="failed to get container status \"deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242\": rpc error: code = NotFound desc = could not find container \"deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242\": container with ID starting with deb6ce5cc5f6a7a23a43a67f54a9bcc3875e91ad84756cf8ac641740ec44d242 not found: ID does not exist"
Sep 16 20:30:00.658578 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.658526    2574 scope.go:117] "RemoveContainer" containerID="f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5"
Sep 16 20:30:00.658808 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:30:00.658791    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5\": container with ID starting with f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5 not found: ID does not exist" containerID="f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5"
Sep 16 20:30:00.658849 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.658813    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5"} err="failed to get container status \"f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5\": rpc error: code = NotFound desc = could not find container \"f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5\": container with ID starting with f23a6d3a3d3a1d05981353587eae04e536f32b32cf5cd402390cc8346d3f95f5 not found: ID does not exist"
Sep 16 20:30:00.668302 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.668275    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"]
Sep 16 20:30:00.675595 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:00.675571    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-699c764795-sgmvw"]
Sep 16 20:30:01.089740 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:01.089711    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" path="/var/lib/kubelet/pods/b2ea18d8-8a9b-4062-ad7e-f2aefee63190/volumes"
Sep 16 20:30:03.245261 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:03.245211    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:03.584363 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:03.584327    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:08.552381 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:08.552343    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:12.098291 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:12.098248    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:16.470011 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:16.469978    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:16.578256 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:16.578221    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:16.673047 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:16.673009    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:17.510286 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:17.510246    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:22.891286 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:22.891183    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:23.737974 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:23.737932    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:28.701748 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:28.701709    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:28.806097 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:28.806062    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:34.620398 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620361    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"]
Sep 16 20:30:34.620799 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620717    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy"
Sep 16 20:30:34.620799 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620729    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy"
Sep 16 20:30:34.620799 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620749    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" containerName="main"
Sep 16 20:30:34.620799 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620754    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" containerName="main"
Sep 16 20:30:34.620936 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620808    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy"
Sep 16 20:30:34.620936 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620818    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bbf6b5-3531-47c6-b6c8-59dcaf30b94b" containerName="main"
Sep 16 20:30:34.620936 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620823    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy"
Sep 16 20:30:34.620936 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620882    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy"
Sep 16 20:30:34.620936 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.620888    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ea18d8-8a9b-4062-ad7e-f2aefee63190" containerName="istio-proxy"
Sep 16 20:30:34.625718 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.625699    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.628185 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.628160    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 20:30:34.628533 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.628514    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:30:34.628712 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.628688    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 20:30:34.628783 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.628735    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-8n4ks\""
Sep 16 20:30:34.628838 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.628735    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 20:30:34.628838 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.628741    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 20:30:34.637875 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.637847    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"]
Sep 16 20:30:34.689785 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.689749    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:34.783546 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.783510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.783765 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.783553    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.783765 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.783582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.783765 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.783627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8223347c-54c9-4106-8a9e-8765fa7d38f0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.783765 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.783657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.783765 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.783688    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.783765 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.783731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8223347c-54c9-4106-8a9e-8765fa7d38f0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.784116 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.783814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzw7d\" (UniqueName: \"kubernetes.io/projected/8223347c-54c9-4106-8a9e-8765fa7d38f0-kube-api-access-wzw7d\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885348 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885348 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8223347c-54c9-4106-8a9e-8765fa7d38f0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885620 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzw7d\" (UniqueName: \"kubernetes.io/projected/8223347c-54c9-4106-8a9e-8765fa7d38f0-kube-api-access-wzw7d\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885620 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885620 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885620 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885620 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8223347c-54c9-4106-8a9e-8765fa7d38f0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885620 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885571    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885934 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885728    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885934 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885835    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.885934 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.886079 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.885973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.886454 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.886427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8223347c-54c9-4106-8a9e-8765fa7d38f0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.888150 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.888124    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.888330 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.888304    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:34.888468 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.888450    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8223347c-54c9-4106-8a9e-8765fa7d38f0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.903377 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.903351    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzw7d\" (UniqueName: \"kubernetes.io/projected/8223347c-54c9-4106-8a9e-8765fa7d38f0-kube-api-access-wzw7d\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:34.937448 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:34.937361    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:35.222964 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:35.222934    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"]
Sep 16 20:30:35.224942 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:30:35.224919    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8223347c_54c9_4106_8a9e_8765fa7d38f0.slice/crio-6df7d6bb0827f7fcbfbbc8adfc2404d3e243cd2d9a034cf61faba13c18f9f055 WatchSource:0}: Error finding container 6df7d6bb0827f7fcbfbbc8adfc2404d3e243cd2d9a034cf61faba13c18f9f055: Status 404 returned error can't find the container with id 6df7d6bb0827f7fcbfbbc8adfc2404d3e243cd2d9a034cf61faba13c18f9f055
Sep 16 20:30:35.835268 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:35.835211    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" event={"ID":"8223347c-54c9-4106-8a9e-8765fa7d38f0","Type":"ContainerStarted","Data":"d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df"}
Sep 16 20:30:35.835803 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:35.835299    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" event={"ID":"8223347c-54c9-4106-8a9e-8765fa7d38f0","Type":"ContainerStarted","Data":"6df7d6bb0827f7fcbfbbc8adfc2404d3e243cd2d9a034cf61faba13c18f9f055"}
Sep 16 20:30:38.854010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:38.853976    2574 generic.go:356] "Generic (PLEG): container finished" podID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerID="d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df" exitCode=0
Sep 16 20:30:38.854425 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:38.854051    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" event={"ID":"8223347c-54c9-4106-8a9e-8765fa7d38f0","Type":"ContainerDied","Data":"d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df"}
Sep 16 20:30:39.861349 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:39.861314    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" event={"ID":"8223347c-54c9-4106-8a9e-8765fa7d38f0","Type":"ContainerStarted","Data":"28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2"}
Sep 16 20:30:39.885503 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:39.885447    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" podStartSLOduration=2.846296055 podStartE2EDuration="5.885426187s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.039130132s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:30:39.882308294 +0000 UTC m=+3071.439653613" watchObservedRunningTime="2026-09-16 20:30:39.885426187 +0000 UTC m=+3071.442771274"
Sep 16 20:30:39.937890 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:39.937832    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:42.879568 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:42.879524    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:44.937739 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:44.937701    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:30:44.939416 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:44.939372    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 16 20:30:46.210513 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:46.210465    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:48.184820 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:48.184776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:48.243815 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:48.243770    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:30:54.938270 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:30:54.938222    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 16 20:31:04.938207 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:04.938161    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 16 20:31:14.937900 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:14.937850    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 16 20:31:24.938809 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:24.938759    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 16 20:31:34.938364 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:34.938317    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 16 20:31:44.938341 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:44.938294    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 16 20:31:54.948684 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:54.948600    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:31:54.962256 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:54.962221    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:31:57.303691 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:57.303656    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:31:57.311299 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:57.310350    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"]
Sep 16 20:31:57.360333 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:31:57.360298    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:09.028528 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:09.028334    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:09.125547 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:09.125510    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:12.333836 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.333777    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main" containerID="cri-o://28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2" gracePeriod=45
Sep 16 20:32:12.896041 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.896015    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:32:12.967583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.967501    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-dshm\") pod \"8223347c-54c9-4106-8a9e-8765fa7d38f0\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") "
Sep 16 20:32:12.967583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.967541    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-home\") pod \"8223347c-54c9-4106-8a9e-8765fa7d38f0\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") "
Sep 16 20:32:12.967813 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.967592    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzw7d\" (UniqueName: \"kubernetes.io/projected/8223347c-54c9-4106-8a9e-8765fa7d38f0-kube-api-access-wzw7d\") pod \"8223347c-54c9-4106-8a9e-8765fa7d38f0\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") "
Sep 16 20:32:12.967813 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.967614    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8223347c-54c9-4106-8a9e-8765fa7d38f0-cabundle-cert\") pod \"8223347c-54c9-4106-8a9e-8765fa7d38f0\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") "
Sep 16 20:32:12.967813 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.967643    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8223347c-54c9-4106-8a9e-8765fa7d38f0-tls-certs\") pod \"8223347c-54c9-4106-8a9e-8765fa7d38f0\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") "
Sep 16 20:32:12.967813 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.967663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-tmp-dir\") pod \"8223347c-54c9-4106-8a9e-8765fa7d38f0\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") "
Sep 16 20:32:12.967813 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.967681    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-model-cache\") pod \"8223347c-54c9-4106-8a9e-8765fa7d38f0\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") "
Sep 16 20:32:12.967813 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.967744    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-kserve-provision-location\") pod \"8223347c-54c9-4106-8a9e-8765fa7d38f0\" (UID: \"8223347c-54c9-4106-8a9e-8765fa7d38f0\") "
Sep 16 20:32:12.968141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.968048    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-home" (OuterVolumeSpecName: "home") pod "8223347c-54c9-4106-8a9e-8765fa7d38f0" (UID: "8223347c-54c9-4106-8a9e-8765fa7d38f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:12.968141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.968128    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "8223347c-54c9-4106-8a9e-8765fa7d38f0" (UID: "8223347c-54c9-4106-8a9e-8765fa7d38f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:12.968252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.968228    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8223347c-54c9-4106-8a9e-8765fa7d38f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8223347c-54c9-4106-8a9e-8765fa7d38f0" (UID: "8223347c-54c9-4106-8a9e-8765fa7d38f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:32:12.968252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.968236    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8223347c-54c9-4106-8a9e-8765fa7d38f0" (UID: "8223347c-54c9-4106-8a9e-8765fa7d38f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:12.970002 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.969976    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8223347c-54c9-4106-8a9e-8765fa7d38f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8223347c-54c9-4106-8a9e-8765fa7d38f0" (UID: "8223347c-54c9-4106-8a9e-8765fa7d38f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:32:12.970151 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.970129    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-dshm" (OuterVolumeSpecName: "dshm") pod "8223347c-54c9-4106-8a9e-8765fa7d38f0" (UID: "8223347c-54c9-4106-8a9e-8765fa7d38f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:12.970234 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.970218    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8223347c-54c9-4106-8a9e-8765fa7d38f0-kube-api-access-wzw7d" (OuterVolumeSpecName: "kube-api-access-wzw7d") pod "8223347c-54c9-4106-8a9e-8765fa7d38f0" (UID: "8223347c-54c9-4106-8a9e-8765fa7d38f0"). InnerVolumeSpecName "kube-api-access-wzw7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:32:12.988225 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:12.988198    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8223347c-54c9-4106-8a9e-8765fa7d38f0" (UID: "8223347c-54c9-4106-8a9e-8765fa7d38f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:32:13.068755 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.068724    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:32:13.068755 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.068752    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:32:13.069335 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.068764    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzw7d\" (UniqueName: \"kubernetes.io/projected/8223347c-54c9-4106-8a9e-8765fa7d38f0-kube-api-access-wzw7d\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:32:13.069335 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.068777    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8223347c-54c9-4106-8a9e-8765fa7d38f0-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:32:13.069335 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.068786    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8223347c-54c9-4106-8a9e-8765fa7d38f0-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:32:13.069335 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.068795    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:32:13.069335 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.068802    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:32:13.069335 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.068811    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8223347c-54c9-4106-8a9e-8765fa7d38f0-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:32:13.390942 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.390899    2574 generic.go:356] "Generic (PLEG): container finished" podID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerID="28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2" exitCode=0
Sep 16 20:32:13.391353 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.390965    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"
Sep 16 20:32:13.391353 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.390962    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" event={"ID":"8223347c-54c9-4106-8a9e-8765fa7d38f0","Type":"ContainerDied","Data":"28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2"}
Sep 16 20:32:13.391353 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.391123    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz" event={"ID":"8223347c-54c9-4106-8a9e-8765fa7d38f0","Type":"ContainerDied","Data":"6df7d6bb0827f7fcbfbbc8adfc2404d3e243cd2d9a034cf61faba13c18f9f055"}
Sep 16 20:32:13.391353 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.391153    2574 scope.go:117] "RemoveContainer" containerID="28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2"
Sep 16 20:32:13.400568 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.400550    2574 scope.go:117] "RemoveContainer" containerID="d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df"
Sep 16 20:32:13.409737 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.409635    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"]
Sep 16 20:32:13.409794 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.409759    2574 scope.go:117] "RemoveContainer" containerID="28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2"
Sep 16 20:32:13.410075 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:32:13.410051    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2\": container with ID starting with 28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2 not found: ID does not exist" containerID="28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2"
Sep 16 20:32:13.410123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.410083    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2"} err="failed to get container status \"28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2\": rpc error: code = NotFound desc = could not find container \"28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2\": container with ID starting with 28900a33a95415f3ac48d7988cbc24a737b696ba2cb45ef16e2ecefd5c88b5e2 not found: ID does not exist"
Sep 16 20:32:13.410123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.410101    2574 scope.go:117] "RemoveContainer" containerID="d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df"
Sep 16 20:32:13.410359 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:32:13.410339    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df\": container with ID starting with d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df not found: ID does not exist" containerID="d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df"
Sep 16 20:32:13.410429 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.410364    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df"} err="failed to get container status \"d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df\": rpc error: code = NotFound desc = could not find container \"d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df\": container with ID starting with d25edd8aab5fd49fe6458f85a615199f07fd72145393b8b0f6f6a577a8dc20df not found: ID does not exist"
Sep 16 20:32:13.415389 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:13.415366    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7ddc8df7cc-z6ppz"]
Sep 16 20:32:15.088519 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:15.088487    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" path="/var/lib/kubelet/pods/8223347c-54c9-4106-8a9e-8765fa7d38f0/volumes"
Sep 16 20:32:46.617296 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:46.617259    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:46.723817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:46.723780    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:46.828149 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:46.828118    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:48.962651 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:48.962616    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:49.071145 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:49.071112    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:49.167636 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:49.167598    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:50.795942 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:50.795912    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:53.691995 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:53.691958    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:55.129484 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:55.129450    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:55.185687 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:55.185651    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:58.953666 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:58.953623    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:32:59.054744 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:32:59.054711    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:00.132685 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:00.132656    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:00.228593 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:00.228558    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:01.718527 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:01.718483    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:02.061114 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:02.061025    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:04.528508 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:04.528465    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:04.591636 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:04.591597    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:05.607330 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:05.607286    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:05.715495 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:05.715459    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:05.815832 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:05.815783    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:06.975025 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:06.974977    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:09.944534 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:09.944497    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:10.054897 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:10.054859    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:10.146244 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:10.146208    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:10.679031 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:10.678991    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:12.889852 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:12.889802    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:12.990270 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:12.990230    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:14.897026 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:14.896968    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:15.135208 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:15.135173    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:15.233762 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:15.233660    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:16.547477 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:16.547442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:16.601190 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:16.601146    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:20.006068 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:20.006025    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:20.103294 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:20.103259    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:21.658067 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:21.657971    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:21.716419 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:21.716371    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:26.150964 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:26.150922    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:26.205635 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:26.205600    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:26.289784 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:26.289741    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:26.347760 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:26.347709    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:27.928698 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:27.928662    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:28.213018 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:28.212930    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:40.188022 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:40.187976    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:51.918174 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.918138    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"]
Sep 16 20:33:51.918633 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.918511    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main"
Sep 16 20:33:51.918633 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.918524    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main"
Sep 16 20:33:51.918633 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.918537    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="storage-initializer"
Sep 16 20:33:51.918633 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.918542    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="storage-initializer"
Sep 16 20:33:51.918633 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.918588    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8223347c-54c9-4106-8a9e-8765fa7d38f0" containerName="main"
Sep 16 20:33:51.921775 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.921755    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:51.924381 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.924358    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 20:33:51.924565 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.924541    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 20:33:51.924669 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.924466    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 20:33:51.924669 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.924495    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-pk8rp\""
Sep 16 20:33:51.924799 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.924549    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 20:33:51.924799 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.924466    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:33:51.937026 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.937002    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"]
Sep 16 20:33:51.945257 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:51.945218    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:52.013786 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.013749    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-home\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.013786 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.013790    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-dshm\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.014012 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.013810    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e1f915-6489-4bf2-8bce-766717f794d4-tls-certs\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.014012 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.013868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-kserve-provision-location\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.014012 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.013946    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47bf5\" (UniqueName: \"kubernetes.io/projected/b1e1f915-6489-4bf2-8bce-766717f794d4-kube-api-access-47bf5\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.014012 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.013970    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-model-cache\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.014012 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.013992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-tmp-dir\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.014181 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.014026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e1f915-6489-4bf2-8bce-766717f794d4-cabundle-cert\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.115276 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.115232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e1f915-6489-4bf2-8bce-766717f794d4-cabundle-cert\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.115489 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.115301    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-home\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.115489 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.115320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-dshm\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.115489 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.115341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e1f915-6489-4bf2-8bce-766717f794d4-tls-certs\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.115489 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.115357    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-kserve-provision-location\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.115489 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.115393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47bf5\" (UniqueName: \"kubernetes.io/projected/b1e1f915-6489-4bf2-8bce-766717f794d4-kube-api-access-47bf5\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.115770 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.115497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-model-cache\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.115770 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.115550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-tmp-dir\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.115893 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.115837    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-kserve-provision-location\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.116052 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.116029    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-tmp-dir\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.116121 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.116066    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-model-cache\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.116121 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.116086    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e1f915-6489-4bf2-8bce-766717f794d4-cabundle-cert\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.116121 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.116094    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-home\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.117823 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.117799    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-dshm\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.117966 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.117950    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e1f915-6489-4bf2-8bce-766717f794d4-tls-certs\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.123757 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.123737    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47bf5\" (UniqueName: \"kubernetes.io/projected/b1e1f915-6489-4bf2-8bce-766717f794d4-kube-api-access-47bf5\") pod \"stop-feature-test-kserve-6c84577c89-sb24m\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.198719 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.198622    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:33:52.232823 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.232773    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:33:52.516370 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.516345    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"]
Sep 16 20:33:52.518174 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:33:52.518133    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e1f915_6489_4bf2_8bce_766717f794d4.slice/crio-72250163fb565986e22d44dfbc3f4f66e92e3997edc74191f4295048ce450950 WatchSource:0}: Error finding container 72250163fb565986e22d44dfbc3f4f66e92e3997edc74191f4295048ce450950: Status 404 returned error can't find the container with id 72250163fb565986e22d44dfbc3f4f66e92e3997edc74191f4295048ce450950
Sep 16 20:33:52.914263 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.914225    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" event={"ID":"b1e1f915-6489-4bf2-8bce-766717f794d4","Type":"ContainerStarted","Data":"d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e"}
Sep 16 20:33:52.914457 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:52.914282    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" event={"ID":"b1e1f915-6489-4bf2-8bce-766717f794d4","Type":"ContainerStarted","Data":"72250163fb565986e22d44dfbc3f4f66e92e3997edc74191f4295048ce450950"}
Sep 16 20:33:54.925758 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:54.925718    2574 generic.go:356] "Generic (PLEG): container finished" podID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerID="d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e" exitCode=0
Sep 16 20:33:54.926160 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:54.925787    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" event={"ID":"b1e1f915-6489-4bf2-8bce-766717f794d4","Type":"ContainerDied","Data":"d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e"}
Sep 16 20:33:55.932951 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:55.932910    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" event={"ID":"b1e1f915-6489-4bf2-8bce-766717f794d4","Type":"ContainerStarted","Data":"d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935"}
Sep 16 20:33:55.957871 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:55.957807    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podStartSLOduration=2.653803062 podStartE2EDuration="4.957792284s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.303989222s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:33:55.955232579 +0000 UTC m=+3267.512577707" watchObservedRunningTime="2026-09-16 20:33:55.957792284 +0000 UTC m=+3267.515137371"
Sep 16 20:33:56.233204 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:33:56.233116    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:34:02.233826 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:02.233786    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:34:02.235714 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:02.235684    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:34:03.243789 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:03.243745    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:34:12.234117 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:12.234064    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:34:18.672862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:18.672820    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:34:21.209538 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:21.209497    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:34:22.233166 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:22.233124    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:34:29.198688 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:29.198658    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:34:29.203703 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:29.203679    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:34:31.585263 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:31.585223    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:34:32.233817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:32.233773    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:34:42.233230 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:42.233185    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:34:52.233389 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:34:52.233295    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:35:02.233520 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:02.233465    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:35:12.234010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:12.233970    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:35:22.234047 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:22.234002    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:35:32.234153 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:32.234094    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:35:42.233557 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:42.233508    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:35:43.333459 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:43.333421    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:35:43.437100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:43.437061    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:35:43.528517 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:43.528475    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:35:52.244039 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:52.244009    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:35:52.253208 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:52.253181    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:35:53.906161 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:53.906126    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"]
Sep 16 20:35:54.667134 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:54.667100    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:35:54.728956 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:35:54.728901    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:06.443873 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:06.443805    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:06.712725 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:06.712623    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:08.929989 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:08.929942    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" containerID="cri-o://d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935" gracePeriod=45
Sep 16 20:36:09.245831 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.245729    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 16 20:36:09.676126 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.676102    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:36:09.691716 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.691690    2574 generic.go:356] "Generic (PLEG): container finished" podID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerID="d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935" exitCode=0
Sep 16 20:36:09.691840 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.691739    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"
Sep 16 20:36:09.691840 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.691781    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" event={"ID":"b1e1f915-6489-4bf2-8bce-766717f794d4","Type":"ContainerDied","Data":"d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935"}
Sep 16 20:36:09.691840 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.691834    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m" event={"ID":"b1e1f915-6489-4bf2-8bce-766717f794d4","Type":"ContainerDied","Data":"72250163fb565986e22d44dfbc3f4f66e92e3997edc74191f4295048ce450950"}
Sep 16 20:36:09.691943 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.691857    2574 scope.go:117] "RemoveContainer" containerID="d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935"
Sep 16 20:36:09.701666 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.701647    2574 scope.go:117] "RemoveContainer" containerID="d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e"
Sep 16 20:36:09.712096 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.712070    2574 scope.go:117] "RemoveContainer" containerID="d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935"
Sep 16 20:36:09.712507 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:36:09.712480    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935\": container with ID starting with d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935 not found: ID does not exist" containerID="d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935"
Sep 16 20:36:09.712581 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.712520    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935"} err="failed to get container status \"d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935\": rpc error: code = NotFound desc = could not find container \"d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935\": container with ID starting with d42185a095b5217cde7be31c28f8bd64b8ec0d1cfc5089b583c05fbca7217935 not found: ID does not exist"
Sep 16 20:36:09.712581 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.712546    2574 scope.go:117] "RemoveContainer" containerID="d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e"
Sep 16 20:36:09.712877 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:36:09.712858    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e\": container with ID starting with d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e not found: ID does not exist" containerID="d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e"
Sep 16 20:36:09.712943 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.712884    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e"} err="failed to get container status \"d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e\": rpc error: code = NotFound desc = could not find container \"d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e\": container with ID starting with d8de652815c500ef30ab0acc0c7fee3da207f73bfa4065e32b2b08071e1c603e not found: ID does not exist"
Sep 16 20:36:09.768033 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768005    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47bf5\" (UniqueName: \"kubernetes.io/projected/b1e1f915-6489-4bf2-8bce-766717f794d4-kube-api-access-47bf5\") pod \"b1e1f915-6489-4bf2-8bce-766717f794d4\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") "
Sep 16 20:36:09.768222 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768058    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-home\") pod \"b1e1f915-6489-4bf2-8bce-766717f794d4\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") "
Sep 16 20:36:09.768222 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768084    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-tmp-dir\") pod \"b1e1f915-6489-4bf2-8bce-766717f794d4\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") "
Sep 16 20:36:09.768222 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768182    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-model-cache\") pod \"b1e1f915-6489-4bf2-8bce-766717f794d4\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") "
Sep 16 20:36:09.768421 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768247    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-dshm\") pod \"b1e1f915-6489-4bf2-8bce-766717f794d4\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") "
Sep 16 20:36:09.768421 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768276    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e1f915-6489-4bf2-8bce-766717f794d4-tls-certs\") pod \"b1e1f915-6489-4bf2-8bce-766717f794d4\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") "
Sep 16 20:36:09.768421 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768306    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e1f915-6489-4bf2-8bce-766717f794d4-cabundle-cert\") pod \"b1e1f915-6489-4bf2-8bce-766717f794d4\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") "
Sep 16 20:36:09.768421 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768345    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-kserve-provision-location\") pod \"b1e1f915-6489-4bf2-8bce-766717f794d4\" (UID: \"b1e1f915-6489-4bf2-8bce-766717f794d4\") "
Sep 16 20:36:09.768640 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768521    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1e1f915-6489-4bf2-8bce-766717f794d4" (UID: "b1e1f915-6489-4bf2-8bce-766717f794d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:36:09.768697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768653    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:36:09.768967 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.768940    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-home" (OuterVolumeSpecName: "home") pod "b1e1f915-6489-4bf2-8bce-766717f794d4" (UID: "b1e1f915-6489-4bf2-8bce-766717f794d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:36:09.769053 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.769004    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e1f915-6489-4bf2-8bce-766717f794d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1e1f915-6489-4bf2-8bce-766717f794d4" (UID: "b1e1f915-6489-4bf2-8bce-766717f794d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:36:09.770720 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.770692    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e1f915-6489-4bf2-8bce-766717f794d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1e1f915-6489-4bf2-8bce-766717f794d4" (UID: "b1e1f915-6489-4bf2-8bce-766717f794d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:36:09.770828 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.770795    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-dshm" (OuterVolumeSpecName: "dshm") pod "b1e1f915-6489-4bf2-8bce-766717f794d4" (UID: "b1e1f915-6489-4bf2-8bce-766717f794d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:36:09.770874 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.770823    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e1f915-6489-4bf2-8bce-766717f794d4-kube-api-access-47bf5" (OuterVolumeSpecName: "kube-api-access-47bf5") pod "b1e1f915-6489-4bf2-8bce-766717f794d4" (UID: "b1e1f915-6489-4bf2-8bce-766717f794d4"). InnerVolumeSpecName "kube-api-access-47bf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:36:09.779344 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.779319    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1e1f915-6489-4bf2-8bce-766717f794d4" (UID: "b1e1f915-6489-4bf2-8bce-766717f794d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:36:09.796159 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.796104    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1e1f915-6489-4bf2-8bce-766717f794d4" (UID: "b1e1f915-6489-4bf2-8bce-766717f794d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:36:09.869113 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.869073    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:36:09.869113 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.869107    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e1f915-6489-4bf2-8bce-766717f794d4-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:36:09.869113 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.869116    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e1f915-6489-4bf2-8bce-766717f794d4-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:36:09.869351 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.869125    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:36:09.869351 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.869134    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47bf5\" (UniqueName: \"kubernetes.io/projected/b1e1f915-6489-4bf2-8bce-766717f794d4-kube-api-access-47bf5\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:36:09.869351 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.869142    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:36:09.869351 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:09.869150    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e1f915-6489-4bf2-8bce-766717f794d4-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:36:10.015869 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:10.015838    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"]
Sep 16 20:36:10.020557 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:10.020531    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6c84577c89-sb24m"]
Sep 16 20:36:11.088905 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:11.088873    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" path="/var/lib/kubelet/pods/b1e1f915-6489-4bf2-8bce-766717f794d4/volumes"
Sep 16 20:36:14.223260 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:14.223213    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:33.243580 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:33.243540    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:33.348060 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:33.348020    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:39.475594 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:39.475555    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:45.261339 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:45.261305    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:45.317066 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:45.317031    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:46.710271 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.710233    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"]
Sep 16 20:36:46.710689 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.710660    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="storage-initializer"
Sep 16 20:36:46.710689 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.710677    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="storage-initializer"
Sep 16 20:36:46.710768 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.710691    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main"
Sep 16 20:36:46.710768 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.710697    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main"
Sep 16 20:36:46.710832 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.710771    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e1f915-6489-4bf2-8bce-766717f794d4" containerName="main"
Sep 16 20:36:46.714239 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.714216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.716829 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.716805    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 16 20:36:46.716962 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.716858    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:36:46.716962 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.716871    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 20:36:46.716962 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.716933    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 20:36:46.717152 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.716958    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 20:36:46.717212 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.717198    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-g9hhn\""
Sep 16 20:36:46.727264 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.727235    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"]
Sep 16 20:36:46.779721 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.779689    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:46.803833 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.803808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.803991 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.803845    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tls-certs\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.803991 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.803873    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tmp-dir\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.803991 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.803902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.803991 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.803925    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-dshm\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.803991 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.803943    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lv22\" (UniqueName: \"kubernetes.io/projected/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kube-api-access-8lv22\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.803991 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.803973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-home\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.803991 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.803992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-model-cache\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.905395 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.905346    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-home\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.905619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.905432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-model-cache\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.905619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.905518    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.905619 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.905574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tls-certs\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.905796 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.905634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tmp-dir\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.905796 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.905676    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.905796 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.905708    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-dshm\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.905796 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.905738    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lv22\" (UniqueName: \"kubernetes.io/projected/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kube-api-access-8lv22\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.906558 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.906534    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-home\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.906819 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.906799    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-model-cache\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.907077 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.907055    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.908701 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.908132    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.908701 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.908386    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tmp-dir\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.912388 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.911171    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-dshm\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.913181 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.913160    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tls-certs\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:46.915115 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:46.915096    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lv22\" (UniqueName: \"kubernetes.io/projected/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kube-api-access-8lv22\") pod \"custom-route-timeout-test-kserve-7db79ff9b7-hh9hs\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:47.024100 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:47.023992    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:47.061060 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:47.060974    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:47.307016 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:47.306876    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"]
Sep 16 20:36:47.309873 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:36:47.309829    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9e2bbe_f2fe_4dcd_aea5_9bbb909f6fd9.slice/crio-8b89f02e384bf1c071cbe89f8d131854c0c8da99346d58717b823220e6ec1b65 WatchSource:0}: Error finding container 8b89f02e384bf1c071cbe89f8d131854c0c8da99346d58717b823220e6ec1b65: Status 404 returned error can't find the container with id 8b89f02e384bf1c071cbe89f8d131854c0c8da99346d58717b823220e6ec1b65
Sep 16 20:36:47.891297 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:47.891256    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" event={"ID":"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9","Type":"ContainerStarted","Data":"828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9"}
Sep 16 20:36:47.891703 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:47.891322    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" event={"ID":"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9","Type":"ContainerStarted","Data":"8b89f02e384bf1c071cbe89f8d131854c0c8da99346d58717b823220e6ec1b65"}
Sep 16 20:36:49.904501 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:49.904469    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerID="828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9" exitCode=0
Sep 16 20:36:49.904906 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:49.904523    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" event={"ID":"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9","Type":"ContainerDied","Data":"828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9"}
Sep 16 20:36:50.914585 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:50.914543    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" event={"ID":"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9","Type":"ContainerStarted","Data":"aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f"}
Sep 16 20:36:50.942367 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:50.942310    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podStartSLOduration=2.648967609 podStartE2EDuration="4.942292658s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.293325049s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:36:50.939317456 +0000 UTC m=+3442.496662565" watchObservedRunningTime="2026-09-16 20:36:50.942292658 +0000 UTC m=+3442.499637744"
Sep 16 20:36:50.968290 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:50.968251    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:36:51.024257 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:51.024213    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:57.024763 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:57.024718    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:36:57.026420 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:36:57.026370    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:37:07.025015 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:37:07.024964    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:37:17.024600 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:37:17.024547    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:37:27.024659 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:37:27.024608    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:37:37.024928 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:37:37.024877    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:37:47.024769 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:37:47.024676    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:37:57.025492 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:37:57.025437    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:38:07.024613 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:07.024567    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:38:08.216251 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:08.216206    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:38:08.321262 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:08.321220    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:38:17.024654 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:17.024608    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:38:20.760016 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:20.759973    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:38:21.153484 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:21.153446    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:38:21.206011 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:21.205971    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:38:27.025396 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:27.025340    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:38:37.024642 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:37.024600    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:38:47.034609 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:47.034578    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:38:47.042323 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:47.042299    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:38:52.696484 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:52.696447    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:38:52.720198 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:52.720161    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"]
Sep 16 20:38:52.795264 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:52.795224    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:38:58.009462 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:58.009420    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:38:58.107813 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:58.107776    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:38:58.208235 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:38:58.208194    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:04.332605 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:04.332566    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:07.746480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:07.746434    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" containerID="cri-o://aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f" gracePeriod=45
Sep 16 20:39:08.036003 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.035909    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 16 20:39:08.501235 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.501210    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:39:08.589220 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589187    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-cabundle-cert\") pod \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") "
Sep 16 20:39:08.589434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589234    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-home\") pod \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") "
Sep 16 20:39:08.589434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589260    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lv22\" (UniqueName: \"kubernetes.io/projected/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kube-api-access-8lv22\") pod \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") "
Sep 16 20:39:08.589434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589311    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-model-cache\") pod \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") "
Sep 16 20:39:08.589434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589346    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-dshm\") pod \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") "
Sep 16 20:39:08.589434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589371    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tls-certs\") pod \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") "
Sep 16 20:39:08.589434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589397    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tmp-dir\") pod \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") "
Sep 16 20:39:08.589737 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589464    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kserve-provision-location\") pod \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\" (UID: \"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9\") "
Sep 16 20:39:08.589737 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589588    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" (UID: "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:39:08.589737 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589734    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:39:08.590009 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.589980    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-home" (OuterVolumeSpecName: "home") pod "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" (UID: "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:39:08.590193 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.590169    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" (UID: "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:39:08.591792 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.591764    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kube-api-access-8lv22" (OuterVolumeSpecName: "kube-api-access-8lv22") pod "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" (UID: "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9"). InnerVolumeSpecName "kube-api-access-8lv22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:39:08.591792 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.591776    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-dshm" (OuterVolumeSpecName: "dshm") pod "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" (UID: "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:39:08.591986 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.591963    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" (UID: "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:39:08.601698 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.601673    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" (UID: "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:39:08.617985 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.617960    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" (UID: "ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:39:08.665320 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.665289    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerID="aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f" exitCode=0
Sep 16 20:39:08.665499 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.665335    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"
Sep 16 20:39:08.665499 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.665368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" event={"ID":"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9","Type":"ContainerDied","Data":"aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f"}
Sep 16 20:39:08.665499 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.665457    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs" event={"ID":"ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9","Type":"ContainerDied","Data":"8b89f02e384bf1c071cbe89f8d131854c0c8da99346d58717b823220e6ec1b65"}
Sep 16 20:39:08.665499 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.665488    2574 scope.go:117] "RemoveContainer" containerID="aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f"
Sep 16 20:39:08.675040 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.675019    2574 scope.go:117] "RemoveContainer" containerID="828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9"
Sep 16 20:39:08.682942 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.682924    2574 scope.go:117] "RemoveContainer" containerID="aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f"
Sep 16 20:39:08.683186 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:39:08.683167    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f\": container with ID starting with aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f not found: ID does not exist" containerID="aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f"
Sep 16 20:39:08.683254 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.683193    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f"} err="failed to get container status \"aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f\": rpc error: code = NotFound desc = could not find container \"aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f\": container with ID starting with aece518577ac70ec6e6a1f86a980147e4d79c9d2fc41b1e8962c21c32fc2693f not found: ID does not exist"
Sep 16 20:39:08.683254 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.683210    2574 scope.go:117] "RemoveContainer" containerID="828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9"
Sep 16 20:39:08.683500 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:39:08.683483    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9\": container with ID starting with 828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9 not found: ID does not exist" containerID="828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9"
Sep 16 20:39:08.683548 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.683506    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9"} err="failed to get container status \"828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9\": rpc error: code = NotFound desc = could not find container \"828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9\": container with ID starting with 828f835f7811e106024616878263c3bdeff49b1fcb4a92f9ca7944f4f4b51ad9 not found: ID does not exist"
Sep 16 20:39:08.688159 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.688136    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"]
Sep 16 20:39:08.690105 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.690085    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:39:08.690194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.690108    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:39:08.690194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.690121    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:39:08.690194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.690132    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:39:08.690194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.690145    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:39:08.690194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.690158    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:39:08.690194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.690172    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lv22\" (UniqueName: \"kubernetes.io/projected/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9-kube-api-access-8lv22\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:39:08.694088 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:08.694069    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7db79ff9b7-hh9hs"]
Sep 16 20:39:09.089841 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:09.089812    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" path="/var/lib/kubelet/pods/ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9/volumes"
Sep 16 20:39:10.386042 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:10.386004    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:10.486194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:10.486162    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:12.223306 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.223270    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"]
Sep 16 20:39:12.223830 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.223810    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main"
Sep 16 20:39:12.223892 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.223833    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main"
Sep 16 20:39:12.223892 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.223853    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="storage-initializer"
Sep 16 20:39:12.223892 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.223862    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="storage-initializer"
Sep 16 20:39:12.223993 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.223953    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9e2bbe-f2fe-4dcd-aea5-9bbb909f6fd9" containerName="main"
Sep 16 20:39:12.228723 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.228704    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.231576 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.231555    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 16 20:39:12.231749 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.231733    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-rmk7x\""
Sep 16 20:39:12.231792 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.231782    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 16 20:39:12.232002 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.231978    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 16 20:39:12.232149 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.231979    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 16 20:39:12.232251 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.232236    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:39:12.238718 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.238697    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"]
Sep 16 20:39:12.245094 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.245068    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:12.318303 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.318272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tls-certs\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.318476 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.318310    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-model-cache\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.318476 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.318374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-home\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.318476 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.318390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.318476 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.318445    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-dshm\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.318630 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.318489    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tmp-dir\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.318630 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.318508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bncf\" (UniqueName: \"kubernetes.io/projected/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kube-api-access-7bncf\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.318630 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.318535    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-cabundle-cert\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.419518 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.419485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-cabundle-cert\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.419658 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.419545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tls-certs\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.419658 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.419571    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-model-cache\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.419658 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.419623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-home\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.419658 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.419638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.419879 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.419680    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-dshm\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.419879 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.419703    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tmp-dir\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.419879 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.419718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bncf\" (UniqueName: \"kubernetes.io/projected/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kube-api-access-7bncf\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.420061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.420032    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-home\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.420237 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.420107    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tmp-dir\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.420237 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.420150    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.420327 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.420314    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-model-cache\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.420368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.420336    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-cabundle-cert\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.421893 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.421873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-dshm\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.422286 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.422268    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tls-certs\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.427315 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.427296    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bncf\" (UniqueName: \"kubernetes.io/projected/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kube-api-access-7bncf\") pod \"lora-single-adapter-test-kserve-968db645f-hwxdv\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.540314 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.540209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:12.820687 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:12.820659    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"]
Sep 16 20:39:12.821981 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:39:12.821957    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f50c384_8886_4ce7_85f2_daa59c9aeb4e.slice/crio-066e6114cb7e31336c1d27705a9c9e6df974e77f9b125003bf19a5b6f68e9843 WatchSource:0}: Error finding container 066e6114cb7e31336c1d27705a9c9e6df974e77f9b125003bf19a5b6f68e9843: Status 404 returned error can't find the container with id 066e6114cb7e31336c1d27705a9c9e6df974e77f9b125003bf19a5b6f68e9843
Sep 16 20:39:13.694328 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:13.694290    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" event={"ID":"2f50c384-8886-4ce7-85f2-daa59c9aeb4e","Type":"ContainerStarted","Data":"7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9"}
Sep 16 20:39:13.694850 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:13.694348    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" event={"ID":"2f50c384-8886-4ce7-85f2-daa59c9aeb4e","Type":"ContainerStarted","Data":"066e6114cb7e31336c1d27705a9c9e6df974e77f9b125003bf19a5b6f68e9843"}
Sep 16 20:39:16.710942 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:16.710908    2574 generic.go:356] "Generic (PLEG): container finished" podID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerID="7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9" exitCode=0
Sep 16 20:39:16.711361 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:16.710982    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" event={"ID":"2f50c384-8886-4ce7-85f2-daa59c9aeb4e","Type":"ContainerDied","Data":"7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9"}
Sep 16 20:39:17.718899 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:17.718864    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" event={"ID":"2f50c384-8886-4ce7-85f2-daa59c9aeb4e","Type":"ContainerStarted","Data":"b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf"}
Sep 16 20:39:17.740705 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:17.740650    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podStartSLOduration=2.811642715 podStartE2EDuration="5.740622261s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.928979546s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:39:17.739502837 +0000 UTC m=+3589.296847957" watchObservedRunningTime="2026-09-16 20:39:17.740622261 +0000 UTC m=+3589.297967349"
Sep 16 20:39:18.540913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:18.540868    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:22.541154 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:22.541049    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:39:22.542981 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:22.542948    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 20:39:29.238135 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:29.238102    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:39:29.244136 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:29.244111    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:39:32.541364 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:32.541322    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 20:39:42.541519 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.541467    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 20:39:42.590111 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.590073    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:42.782616 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.782576    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"]
Sep 16 20:39:42.787278 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.787253    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:42.790377 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.790353    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 16 20:39:42.790532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.790437    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 20:39:42.790532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.790437    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-fmnd7\""
Sep 16 20:39:42.790532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.790353    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 20:39:42.800990 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.800966    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"]
Sep 16 20:39:42.906314 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.906275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9959\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-kube-api-access-b9959\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:42.906501 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.906328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-token\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:42.906501 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.906419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-envoy\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:42.906501 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.906476    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-socket\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:42.906624 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.906542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:42.906624 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.906594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:42.906701 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.906645    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-data\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:42.906701 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.906666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-credential-socket\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:42.906701 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:42.906690    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-certs\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.007711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.007673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.007711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.007714    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.007969 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.007738    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-data\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.007969 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.007765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-credential-socket\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.007969 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.007799    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-certs\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.007969 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.007832    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9959\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-kube-api-access-b9959\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.007969 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.007878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-token\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.007969 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.007930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-envoy\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.007969 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.007967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-socket\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.008398 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.008321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-data\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.008585 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.008547    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-certs\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.008684 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.008597    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-credential-socket\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.008909 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.008883    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-socket\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.009000 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.008966    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.010427 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.010391    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-podinfo\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.010608 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.010586    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-envoy\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.015951 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.015930    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-token\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.016055 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.016001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9959\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-kube-api-access-b9959\") pod \"router-gateway-1-openshift-default-7b899fb444-gbsg8\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.099866 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.099832    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:43.379798 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.379771    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"]
Sep 16 20:39:43.382584 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:39:43.382545    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178c1780_1c19_4fa5_8ba8_ba6db5b7e9ff.slice/crio-6da2c0106069a092203e1d9cddbae367c50d8f9bb70229c9f1bb68b01e4bd4c7 WatchSource:0}: Error finding container 6da2c0106069a092203e1d9cddbae367c50d8f9bb70229c9f1bb68b01e4bd4c7: Status 404 returned error can't find the container with id 6da2c0106069a092203e1d9cddbae367c50d8f9bb70229c9f1bb68b01e4bd4c7
Sep 16 20:39:43.385523 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.385483    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:39:43.385649 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.385570    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:39:43.385649 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.385613    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:39:43.713565 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.713456    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:43.864637 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.864592    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" event={"ID":"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff","Type":"ContainerStarted","Data":"73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff"}
Sep 16 20:39:43.864816 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.864664    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" event={"ID":"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff","Type":"ContainerStarted","Data":"6da2c0106069a092203e1d9cddbae367c50d8f9bb70229c9f1bb68b01e4bd4c7"}
Sep 16 20:39:43.885202 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:43.885145    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" podStartSLOduration=1.885123645 podStartE2EDuration="1.885123645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:39:43.883773063 +0000 UTC m=+3615.441118150" watchObservedRunningTime="2026-09-16 20:39:43.885123645 +0000 UTC m=+3615.442468734"
Sep 16 20:39:44.100415 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:44.100373    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:44.101928 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:44.101898    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 16 20:39:45.105420 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:45.105380    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:45.875970 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:45.875935    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:45.877194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:45.877167    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:46.265419 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:46.265302    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:48.924643 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:48.924601    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:52.139148 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:52.139109    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:52.541822 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:52.541711    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 20:39:54.774070 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:54.773971    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:54.834934 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:54.834889    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:54.929300 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:54.929210    2574 generic.go:356] "Generic (PLEG): container finished" podID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerID="73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff" exitCode=0
Sep 16 20:39:54.929453 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:54.929288    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" event={"ID":"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff","Type":"ContainerDied","Data":"73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff"}
Sep 16 20:39:54.929706 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:54.929691    2574 scope.go:117] "RemoveContainer" containerID="73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff"
Sep 16 20:39:54.930775 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:54.930740    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:39:54.930852 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:54.930835    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:39:54.930903 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:54.930889    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:39:55.925415 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:55.925363    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:55.937012 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:55.936977    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" event={"ID":"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff","Type":"ContainerStarted","Data":"9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2"}
Sep 16 20:39:56.100800 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:56.100755    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:39:56.101034 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:56.101008    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 16 20:39:56.485688 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:56.485637    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:56.584010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:56.583972    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:39:57.100350 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:57.100305    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 16 20:39:58.100934 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:58.100885    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 16 20:39:59.101154 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:39:59.101120    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 16 20:40:00.100884 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:00.100833    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 16 20:40:00.875873 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:00.875836    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:40:01.253238 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:01.253133    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:40:01.254890 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:01.254863    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:40:02.541515 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:02.541478    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 20:40:12.541092 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:12.541040    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 20:40:22.540907 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:22.540851    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 20:40:32.550819 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:32.550787    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:40:32.558564 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:32.558539    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:40:59.396597 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:59.396510    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-968db645f-hwxdv_2f50c384-8886-4ce7-85f2-daa59c9aeb4e/storage-initializer/0.log"
Sep 16 20:40:59.407721 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:40:59.407691    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-968db645f-hwxdv_2f50c384-8886-4ce7-85f2-daa59c9aeb4e/main/0.log"
Sep 16 20:41:01.098140 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:01.098103    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"]
Sep 16 20:41:01.111480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:01.111442    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:41:16.123675 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.123627    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main" containerID="cri-o://b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf" gracePeriod=45
Sep 16 20:41:16.552182 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.552083    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 20:41:16.779515 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.779492    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:41:16.798141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798105    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-model-cache\") pod \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") "
Sep 16 20:41:16.798141 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798137    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-home\") pod \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") "
Sep 16 20:41:16.798340 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798156    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-dshm\") pod \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") "
Sep 16 20:41:16.798340 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798211    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kserve-provision-location\") pod \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") "
Sep 16 20:41:16.798340 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798263    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tmp-dir\") pod \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") "
Sep 16 20:41:16.798340 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798293    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tls-certs\") pod \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") "
Sep 16 20:41:16.798585 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798348    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bncf\" (UniqueName: \"kubernetes.io/projected/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kube-api-access-7bncf\") pod \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") "
Sep 16 20:41:16.798585 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798350    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f50c384-8886-4ce7-85f2-daa59c9aeb4e" (UID: "2f50c384-8886-4ce7-85f2-daa59c9aeb4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:41:16.798585 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798386    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-cabundle-cert\") pod \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\" (UID: \"2f50c384-8886-4ce7-85f2-daa59c9aeb4e\") "
Sep 16 20:41:16.798585 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798539    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-home" (OuterVolumeSpecName: "home") pod "2f50c384-8886-4ce7-85f2-daa59c9aeb4e" (UID: "2f50c384-8886-4ce7-85f2-daa59c9aeb4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:41:16.798818 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798795    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:41:16.798876 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798821    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:41:16.798876 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.798870    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f50c384-8886-4ce7-85f2-daa59c9aeb4e" (UID: "2f50c384-8886-4ce7-85f2-daa59c9aeb4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:41:16.801450 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.801395    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-dshm" (OuterVolumeSpecName: "dshm") pod "2f50c384-8886-4ce7-85f2-daa59c9aeb4e" (UID: "2f50c384-8886-4ce7-85f2-daa59c9aeb4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:41:16.803557 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.803493    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kube-api-access-7bncf" (OuterVolumeSpecName: "kube-api-access-7bncf") pod "2f50c384-8886-4ce7-85f2-daa59c9aeb4e" (UID: "2f50c384-8886-4ce7-85f2-daa59c9aeb4e"). InnerVolumeSpecName "kube-api-access-7bncf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:41:16.804246 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.804219    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f50c384-8886-4ce7-85f2-daa59c9aeb4e" (UID: "2f50c384-8886-4ce7-85f2-daa59c9aeb4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:41:16.812788 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.812755    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f50c384-8886-4ce7-85f2-daa59c9aeb4e" (UID: "2f50c384-8886-4ce7-85f2-daa59c9aeb4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:41:16.829674 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.829645    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f50c384-8886-4ce7-85f2-daa59c9aeb4e" (UID: "2f50c384-8886-4ce7-85f2-daa59c9aeb4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:41:16.899860 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.899823    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:41:16.899860 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.899855    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:41:16.899860 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.899868    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:41:16.900108 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.899878    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bncf\" (UniqueName: \"kubernetes.io/projected/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-kube-api-access-7bncf\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:41:16.900108 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.899886    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:41:16.900108 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:16.899895    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f50c384-8886-4ce7-85f2-daa59c9aeb4e-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:41:17.405617 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.405516    2574 generic.go:356] "Generic (PLEG): container finished" podID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerID="b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf" exitCode=0
Sep 16 20:41:17.405617 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.405582    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"
Sep 16 20:41:17.405617 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.405600    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" event={"ID":"2f50c384-8886-4ce7-85f2-daa59c9aeb4e","Type":"ContainerDied","Data":"b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf"}
Sep 16 20:41:17.406195 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.405654    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv" event={"ID":"2f50c384-8886-4ce7-85f2-daa59c9aeb4e","Type":"ContainerDied","Data":"066e6114cb7e31336c1d27705a9c9e6df974e77f9b125003bf19a5b6f68e9843"}
Sep 16 20:41:17.406195 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.405678    2574 scope.go:117] "RemoveContainer" containerID="b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf"
Sep 16 20:41:17.415156 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.415142    2574 scope.go:117] "RemoveContainer" containerID="7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9"
Sep 16 20:41:17.423997 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.423953    2574 scope.go:117] "RemoveContainer" containerID="b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf"
Sep 16 20:41:17.424315 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:41:17.424284    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf\": container with ID starting with b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf not found: ID does not exist" containerID="b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf"
Sep 16 20:41:17.424455 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.424325    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf"} err="failed to get container status \"b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf\": rpc error: code = NotFound desc = could not find container \"b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf\": container with ID starting with b7d670b1da88d2050e9b5bb8db54a54b9d693d0fad10898a39ec58c174025ecf not found: ID does not exist"
Sep 16 20:41:17.424455 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.424345    2574 scope.go:117] "RemoveContainer" containerID="7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9"
Sep 16 20:41:17.424700 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:41:17.424676    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9\": container with ID starting with 7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9 not found: ID does not exist" containerID="7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9"
Sep 16 20:41:17.424763 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.424706    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9"} err="failed to get container status \"7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9\": rpc error: code = NotFound desc = could not find container \"7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9\": container with ID starting with 7cbf96ee38f84a2600a8fb19c7ab175eb301775e0c36039d875aa996d81bf2e9 not found: ID does not exist"
Sep 16 20:41:17.426321 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.426297    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"]
Sep 16 20:41:17.428976 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:17.428957    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-968db645f-hwxdv"]
Sep 16 20:41:19.095325 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:19.095289    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" path="/var/lib/kubelet/pods/2f50c384-8886-4ce7-85f2-daa59c9aeb4e/volumes"
Sep 16 20:41:20.917458 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:20.917397    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:41:21.017161 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:21.017120    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:41:21.117687 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:21.117645    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:41:26.511973 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.511932    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"]
Sep 16 20:41:26.512357 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.512282    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="storage-initializer"
Sep 16 20:41:26.512357 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.512295    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="storage-initializer"
Sep 16 20:41:26.512357 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.512330    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main"
Sep 16 20:41:26.512357 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.512335    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main"
Sep 16 20:41:26.512521 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.512425    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f50c384-8886-4ce7-85f2-daa59c9aeb4e" containerName="main"
Sep 16 20:41:26.517238 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.517220    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.520340 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.520312    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-vdbwd\""
Sep 16 20:41:26.520340 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.520334    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 16 20:41:26.520523 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.520502    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 16 20:41:26.520606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.520588    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:41:26.520691 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.520677    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 16 20:41:26.520810 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.520797    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 16 20:41:26.528713 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.528691    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"]
Sep 16 20:41:26.583626 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.583588    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:41:26.586571 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.586550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg2ll\" (UniqueName: \"kubernetes.io/projected/d0b4f187-8edc-49bf-8112-fff652856580-kube-api-access-lg2ll\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.586673 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.586580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-model-cache\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.586673 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.586604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.586673 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.586640    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-home\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.586673 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.586661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-dshm\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.586837 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.586718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b4f187-8edc-49bf-8112-fff652856580-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.586837 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.586748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.586914 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.586846    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b4f187-8edc-49bf-8112-fff652856580-tls-certs\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687354 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg2ll\" (UniqueName: \"kubernetes.io/projected/d0b4f187-8edc-49bf-8112-fff652856580-kube-api-access-lg2ll\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687354 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687359    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-model-cache\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-home\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-dshm\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687627 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b4f187-8edc-49bf-8112-fff652856580-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687851 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687851 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b4f187-8edc-49bf-8112-fff652856580-tls-certs\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687851 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687765    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.687851 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687813    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-model-cache\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.688025 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687898    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-home\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.688025 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.687995    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.688306 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.688279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b4f187-8edc-49bf-8112-fff652856580-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.689914 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.689894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-dshm\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.690214 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.690196    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b4f187-8edc-49bf-8112-fff652856580-tls-certs\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.702896 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.702875    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg2ll\" (UniqueName: \"kubernetes.io/projected/d0b4f187-8edc-49bf-8112-fff652856580-kube-api-access-lg2ll\") pod \"lora-multiple-adapters-test-kserve-69d764b5b6-8zx47\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:26.829186 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:26.829093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:27.110538 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:27.110514    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"]
Sep 16 20:41:27.111898 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:41:27.111872    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b4f187_8edc_49bf_8112_fff652856580.slice/crio-4235bd037c7e3a5fd9bb9965a41b3be4dcbd99edf5d54c6a90533f61e673a283 WatchSource:0}: Error finding container 4235bd037c7e3a5fd9bb9965a41b3be4dcbd99edf5d54c6a90533f61e673a283: Status 404 returned error can't find the container with id 4235bd037c7e3a5fd9bb9965a41b3be4dcbd99edf5d54c6a90533f61e673a283
Sep 16 20:41:27.461817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:27.461702    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" event={"ID":"d0b4f187-8edc-49bf-8112-fff652856580","Type":"ContainerStarted","Data":"befbfae874f21c1d9e9fd610b55742c8acb7b5b588c4c539798917ae02428772"}
Sep 16 20:41:27.461817 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:27.461795    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" event={"ID":"d0b4f187-8edc-49bf-8112-fff652856580","Type":"ContainerStarted","Data":"4235bd037c7e3a5fd9bb9965a41b3be4dcbd99edf5d54c6a90533f61e673a283"}
Sep 16 20:41:27.838597 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:27.838562    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:41:27.939479 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:27.939437    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:41:30.483027 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:30.482983    2574 generic.go:356] "Generic (PLEG): container finished" podID="d0b4f187-8edc-49bf-8112-fff652856580" containerID="befbfae874f21c1d9e9fd610b55742c8acb7b5b588c4c539798917ae02428772" exitCode=0
Sep 16 20:41:30.483452 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:30.483031    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" event={"ID":"d0b4f187-8edc-49bf-8112-fff652856580","Type":"ContainerDied","Data":"befbfae874f21c1d9e9fd610b55742c8acb7b5b588c4c539798917ae02428772"}
Sep 16 20:41:31.491326 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:31.491289    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" event={"ID":"d0b4f187-8edc-49bf-8112-fff652856580","Type":"ContainerStarted","Data":"30befa76d689be6d8dfd7d4daf6fd13519903a4ce59033ed63b29b94c509cf66"}
Sep 16 20:41:31.518171 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:31.518115    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podStartSLOduration=2.465087104 podStartE2EDuration="5.518097565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.053010461s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:41:31.515512027 +0000 UTC m=+3723.072857135" watchObservedRunningTime="2026-09-16 20:41:31.518097565 +0000 UTC m=+3723.075442653"
Sep 16 20:41:31.830153 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:31.830040    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:36.830052 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:36.830009    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:41:36.831721 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:36.831678    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 20:41:46.830637 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:46.830585    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 20:41:56.829796 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:41:56.829752    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 20:42:01.416524 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:01.416482    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:42:01.509162 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:01.509127    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:42:06.830168 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:06.830115    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 20:42:16.829746 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:16.829700    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 20:42:26.830624 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:26.830526    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 20:42:36.829888 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:36.829840    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 20:42:46.840942 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:46.840909    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:42:46.849003 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:46.848977    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:42:51.471492 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:51.471453    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:42:51.568568 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:51.568528    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:42:56.434664 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:56.434626    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"]
Sep 16 20:42:56.435191 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:56.434927    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy" containerID="cri-o://9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2" gracePeriod=30
Sep 16 20:42:56.487348 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:56.487312    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:42:56.552027 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:56.551981    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:42:56.782471 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:42:56.782425    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:01.256216 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.256167    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 20:43:01.678097 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.678070    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:43:01.751454 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751423    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-data\") pod \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") "
Sep 16 20:43:01.751630 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751472    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-envoy\") pod \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") "
Sep 16 20:43:01.751630 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751498    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9959\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-kube-api-access-b9959\") pod \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") "
Sep 16 20:43:01.751630 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-token\") pod \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") "
Sep 16 20:43:01.751802 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751658    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" (UID: "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:01.751802 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751739    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-certs\") pod \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") "
Sep 16 20:43:01.751802 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751780    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istiod-ca-cert\") pod \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") "
Sep 16 20:43:01.751971 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751822    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-socket\") pod \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") "
Sep 16 20:43:01.751971 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751852    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-podinfo\") pod \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") "
Sep 16 20:43:01.751971 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.751898    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-credential-socket\") pod \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\" (UID: \"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff\") "
Sep 16 20:43:01.752121 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.752040    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" (UID: "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:01.752121 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.752107    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" (UID: "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:01.752233 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.752208    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" (UID: "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:01.752312 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.752292    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-credential-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:01.752378 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.752316    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-data\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:01.752378 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.752331    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:01.752378 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.752344    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-workload-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:01.752625 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.752565    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" (UID: "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:43:01.754222 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.754156    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" (UID: "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:01.754222 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.754159    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" (UID: "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 20:43:01.754511 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.754490    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" (UID: "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:43:01.754585 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.754548    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-kube-api-access-b9959" (OuterVolumeSpecName: "kube-api-access-b9959") pod "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" (UID: "178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff"). InnerVolumeSpecName "kube-api-access-b9959". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:43:01.853015 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.852981    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istiod-ca-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:01.853015 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.853011    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-podinfo\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:01.853209 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.853022    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-envoy\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:01.853209 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.853034    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9959\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-kube-api-access-b9959\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:01.853209 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:01.853048    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff-istio-token\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:02.016313 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.016271    2574 generic.go:356] "Generic (PLEG): container finished" podID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerID="9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2" exitCode=0
Sep 16 20:43:02.016525 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.016318    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"
Sep 16 20:43:02.016525 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.016323    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" event={"ID":"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff","Type":"ContainerDied","Data":"9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2"}
Sep 16 20:43:02.016525 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.016377    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8" event={"ID":"178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff","Type":"ContainerDied","Data":"6da2c0106069a092203e1d9cddbae367c50d8f9bb70229c9f1bb68b01e4bd4c7"}
Sep 16 20:43:02.016525 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.016421    2574 scope.go:117] "RemoveContainer" containerID="9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2"
Sep 16 20:43:02.026844 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.026819    2574 scope.go:117] "RemoveContainer" containerID="73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff"
Sep 16 20:43:02.035317 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.035296    2574 scope.go:117] "RemoveContainer" containerID="9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2"
Sep 16 20:43:02.035629 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:43:02.035608    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2\": container with ID starting with 9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2 not found: ID does not exist" containerID="9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2"
Sep 16 20:43:02.035697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.035639    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2"} err="failed to get container status \"9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2\": rpc error: code = NotFound desc = could not find container \"9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2\": container with ID starting with 9abf69ec833e1b0a0d5fe2d9a89b0ab1e64f685c3e1071b5dc76309f31dbb0b2 not found: ID does not exist"
Sep 16 20:43:02.035697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.035658    2574 scope.go:117] "RemoveContainer" containerID="73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff"
Sep 16 20:43:02.035920 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:43:02.035906    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff\": container with ID starting with 73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff not found: ID does not exist" containerID="73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff"
Sep 16 20:43:02.035963 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.035924    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff"} err="failed to get container status \"73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff\": rpc error: code = NotFound desc = could not find container \"73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff\": container with ID starting with 73ac6f241fb5390c7c54bfc0e26974941ef50781c5543ab1a1857cec6fbd2aff not found: ID does not exist"
Sep 16 20:43:02.041466 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.041391    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"]
Sep 16 20:43:02.044928 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:02.044902    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-7b899fb444-gbsg8"]
Sep 16 20:43:03.089466 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:03.089427    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" path="/var/lib/kubelet/pods/178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff/volumes"
Sep 16 20:43:08.894071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:08.894032    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:15.322249 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:15.322207    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-69d764b5b6-8zx47_d0b4f187-8edc-49bf-8112-fff652856580/storage-initializer/0.log"
Sep 16 20:43:15.334605 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:15.334580    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-69d764b5b6-8zx47_d0b4f187-8edc-49bf-8112-fff652856580/main/0.log"
Sep 16 20:43:16.513117 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:16.513082    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"]
Sep 16 20:43:16.514457 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:16.514412    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:18.510234 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:18.510187    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:19.300514 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:19.300469    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:24.983475 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:24.983440    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:31.535809 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:31.535766    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main" containerID="cri-o://30befa76d689be6d8dfd7d4daf6fd13519903a4ce59033ed63b29b94c509cf66" gracePeriod=45
Sep 16 20:43:31.842366 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:31.842319    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 20:43:32.191793 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.191760    2574 generic.go:356] "Generic (PLEG): container finished" podID="d0b4f187-8edc-49bf-8112-fff652856580" containerID="30befa76d689be6d8dfd7d4daf6fd13519903a4ce59033ed63b29b94c509cf66" exitCode=0
Sep 16 20:43:32.191976 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.191828    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" event={"ID":"d0b4f187-8edc-49bf-8112-fff652856580","Type":"ContainerDied","Data":"30befa76d689be6d8dfd7d4daf6fd13519903a4ce59033ed63b29b94c509cf66"}
Sep 16 20:43:32.191976 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.191925    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47" event={"ID":"d0b4f187-8edc-49bf-8112-fff652856580","Type":"ContainerDied","Data":"4235bd037c7e3a5fd9bb9965a41b3be4dcbd99edf5d54c6a90533f61e673a283"}
Sep 16 20:43:32.191976 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.191950    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4235bd037c7e3a5fd9bb9965a41b3be4dcbd99edf5d54c6a90533f61e673a283"
Sep 16 20:43:32.196222 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.196203    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:43:32.231597 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.231566    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-model-cache\") pod \"d0b4f187-8edc-49bf-8112-fff652856580\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") "
Sep 16 20:43:32.231760 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.231611    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b4f187-8edc-49bf-8112-fff652856580-cabundle-cert\") pod \"d0b4f187-8edc-49bf-8112-fff652856580\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") "
Sep 16 20:43:32.231760 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.231675    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-kserve-provision-location\") pod \"d0b4f187-8edc-49bf-8112-fff652856580\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") "
Sep 16 20:43:32.231760 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.231708    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-tmp-dir\") pod \"d0b4f187-8edc-49bf-8112-fff652856580\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") "
Sep 16 20:43:32.231943 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.231761    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-dshm\") pod \"d0b4f187-8edc-49bf-8112-fff652856580\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") "
Sep 16 20:43:32.231943 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.231821    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0b4f187-8edc-49bf-8112-fff652856580" (UID: "d0b4f187-8edc-49bf-8112-fff652856580"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:32.231943 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.231830    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-home\") pod \"d0b4f187-8edc-49bf-8112-fff652856580\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") "
Sep 16 20:43:32.231943 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.231904    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg2ll\" (UniqueName: \"kubernetes.io/projected/d0b4f187-8edc-49bf-8112-fff652856580-kube-api-access-lg2ll\") pod \"d0b4f187-8edc-49bf-8112-fff652856580\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") "
Sep 16 20:43:32.232165 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.231955    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b4f187-8edc-49bf-8112-fff652856580-tls-certs\") pod \"d0b4f187-8edc-49bf-8112-fff652856580\" (UID: \"d0b4f187-8edc-49bf-8112-fff652856580\") "
Sep 16 20:43:32.232165 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.232038    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b4f187-8edc-49bf-8112-fff652856580-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0b4f187-8edc-49bf-8112-fff652856580" (UID: "d0b4f187-8edc-49bf-8112-fff652856580"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:43:32.232277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.232251    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:32.232277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.232269    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b4f187-8edc-49bf-8112-fff652856580-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:32.232680 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.232653    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-home" (OuterVolumeSpecName: "home") pod "d0b4f187-8edc-49bf-8112-fff652856580" (UID: "d0b4f187-8edc-49bf-8112-fff652856580"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:32.235067 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.235021    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-dshm" (OuterVolumeSpecName: "dshm") pod "d0b4f187-8edc-49bf-8112-fff652856580" (UID: "d0b4f187-8edc-49bf-8112-fff652856580"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:32.235184 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.235156    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b4f187-8edc-49bf-8112-fff652856580-kube-api-access-lg2ll" (OuterVolumeSpecName: "kube-api-access-lg2ll") pod "d0b4f187-8edc-49bf-8112-fff652856580" (UID: "d0b4f187-8edc-49bf-8112-fff652856580"). InnerVolumeSpecName "kube-api-access-lg2ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:43:32.235341 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.235320    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b4f187-8edc-49bf-8112-fff652856580-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0b4f187-8edc-49bf-8112-fff652856580" (UID: "d0b4f187-8edc-49bf-8112-fff652856580"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:43:32.244763 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.244738    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0b4f187-8edc-49bf-8112-fff652856580" (UID: "d0b4f187-8edc-49bf-8112-fff652856580"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:32.257499 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.257467    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0b4f187-8edc-49bf-8112-fff652856580" (UID: "d0b4f187-8edc-49bf-8112-fff652856580"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:43:32.333580 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.333537    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b4f187-8edc-49bf-8112-fff652856580-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:32.333580 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.333577    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:32.333580 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.333588    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:32.333862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.333597    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:32.333862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.333605    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b4f187-8edc-49bf-8112-fff652856580-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:32.333862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:32.333614    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg2ll\" (UniqueName: \"kubernetes.io/projected/d0b4f187-8edc-49bf-8112-fff652856580-kube-api-access-lg2ll\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:33.197744 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:33.197711    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"
Sep 16 20:43:33.224738 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:33.224709    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"]
Sep 16 20:43:33.229045 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:33.229021    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-69d764b5b6-8zx47"]
Sep 16 20:43:35.089267 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:35.089227    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0b4f187-8edc-49bf-8112-fff652856580" path="/var/lib/kubelet/pods/d0b4f187-8edc-49bf-8112-fff652856580/volumes"
Sep 16 20:43:36.525278 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:36.525243    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:36.627651 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:36.627613    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:36.738224 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:36.738188    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:36.840360 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:36.840324    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:36.898643 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:36.898606    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:39.322228 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.322189    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:39.376911 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.376870    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:39.845341 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845304    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"]
Sep 16 20:43:39.845668 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845655    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy"
Sep 16 20:43:39.845668 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845669    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy"
Sep 16 20:43:39.845749 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845678    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main"
Sep 16 20:43:39.845749 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845683    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main"
Sep 16 20:43:39.845749 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845692    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy"
Sep 16 20:43:39.845749 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845698    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy"
Sep 16 20:43:39.845749 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845703    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="storage-initializer"
Sep 16 20:43:39.845749 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845708    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="storage-initializer"
Sep 16 20:43:39.845936 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845767    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy"
Sep 16 20:43:39.845936 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845776    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="178c1780-1c19-4fa5-8ba8-ba6db5b7e9ff" containerName="istio-proxy"
Sep 16 20:43:39.845936 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.845783    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0b4f187-8edc-49bf-8112-fff652856580" containerName="main"
Sep 16 20:43:39.850471 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.850451    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:39.854189 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.854158    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 20:43:39.854189 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.854178    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 20:43:39.854380 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.854195    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 20:43:39.854380 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.854206    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-58p6d\""
Sep 16 20:43:39.856015 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.855991    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"]
Sep 16 20:43:39.894972 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.894945    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4mz\" (UniqueName: \"kubernetes.io/projected/7af0756f-e55a-49f5-838f-031bd412924e-kube-api-access-7s4mz\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:39.912097 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.912067    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:39.996113 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:39.996077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4mz\" (UniqueName: \"kubernetes.io/projected/7af0756f-e55a-49f5-838f-031bd412924e-kube-api-access-7s4mz\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:40.005603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:40.005571    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4mz\" (UniqueName: \"kubernetes.io/projected/7af0756f-e55a-49f5-838f-031bd412924e-kube-api-access-7s4mz\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:40.045574 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:40.045540    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:42.653026 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:42.652990    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:42.752783 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:42.752746    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:48.270775 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.270737    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:48.278577 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.278550    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:48.371678 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.371639    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:48.371876 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.371788    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:48.374579 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.374559    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") DevicePath \"csi-e04fa74e4d9acc33a95c4c8614edc5613e24db7b3712c47f712892001a69dd50\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:48.599227 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.599198    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/97aad42d75204328628c95717971be9e95c5ebdfa53243eec6197da38c3c3715/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:48.612002 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.611975    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"e2e-pvc-lora-download-4c8hb\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:48.722873 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.722833    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:48.773529 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.773485    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:43:48.870467 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:48.870364    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:49.139704 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:49.139680    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"]
Sep 16 20:43:49.141634 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:43:49.141607    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7af0756f_e55a_49f5_838f_031bd412924e.slice/crio-938bbae68b4718b20a025c12524f195e2214c5e0584d347d447093091c580ad0 WatchSource:0}: Error finding container 938bbae68b4718b20a025c12524f195e2214c5e0584d347d447093091c580ad0: Status 404 returned error can't find the container with id 938bbae68b4718b20a025c12524f195e2214c5e0584d347d447093091c580ad0
Sep 16 20:43:49.283906 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:49.283862    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb" event={"ID":"7af0756f-e55a-49f5-838f-031bd412924e","Type":"ContainerStarted","Data":"bf11d69cc0fdc386294d6865f701c1cbc689184cc12012a5b919d1a7733af39f"}
Sep 16 20:43:49.284328 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:49.283920    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb" event={"ID":"7af0756f-e55a-49f5-838f-031bd412924e","Type":"ContainerStarted","Data":"938bbae68b4718b20a025c12524f195e2214c5e0584d347d447093091c580ad0"}
Sep 16 20:43:49.300085 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:49.300032    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb" podStartSLOduration=13.300016934 podStartE2EDuration="13.300016934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:43:49.298053985 +0000 UTC m=+3860.855399084" watchObservedRunningTime="2026-09-16 20:43:49.300016934 +0000 UTC m=+3860.857362059"
Sep 16 20:43:51.300019 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:51.299988    2574 generic.go:356] "Generic (PLEG): container finished" podID="7af0756f-e55a-49f5-838f-031bd412924e" containerID="bf11d69cc0fdc386294d6865f701c1cbc689184cc12012a5b919d1a7733af39f" exitCode=0
Sep 16 20:43:51.300377 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:51.300073    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb" event={"ID":"7af0756f-e55a-49f5-838f-031bd412924e","Type":"ContainerDied","Data":"bf11d69cc0fdc386294d6865f701c1cbc689184cc12012a5b919d1a7733af39f"}
Sep 16 20:43:52.447440 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:52.447397    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:52.505999 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:52.505968    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"7af0756f-e55a-49f5-838f-031bd412924e\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") "
Sep 16 20:43:52.506136 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:52.506034    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s4mz\" (UniqueName: \"kubernetes.io/projected/7af0756f-e55a-49f5-838f-031bd412924e-kube-api-access-7s4mz\") pod \"7af0756f-e55a-49f5-838f-031bd412924e\" (UID: \"7af0756f-e55a-49f5-838f-031bd412924e\") "
Sep 16 20:43:52.508342 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:52.508311    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af0756f-e55a-49f5-838f-031bd412924e-kube-api-access-7s4mz" (OuterVolumeSpecName: "kube-api-access-7s4mz") pod "7af0756f-e55a-49f5-838f-031bd412924e" (UID: "7af0756f-e55a-49f5-838f-031bd412924e"). InnerVolumeSpecName "kube-api-access-7s4mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:43:52.510729 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:52.510705    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59" (OuterVolumeSpecName: "model-storage") pod "7af0756f-e55a-49f5-838f-031bd412924e" (UID: "7af0756f-e55a-49f5-838f-031bd412924e"). InnerVolumeSpecName "pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 20:43:52.607234 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:52.607205    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") on node \"ip-10-0-141-26.ec2.internal\" "
Sep 16 20:43:52.607234 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:52.607234    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s4mz\" (UniqueName: \"kubernetes.io/projected/7af0756f-e55a-49f5-838f-031bd412924e-kube-api-access-7s4mz\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:43:53.313347 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:53.313315    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"
Sep 16 20:43:53.313546 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:53.313346    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb" event={"ID":"7af0756f-e55a-49f5-838f-031bd412924e","Type":"ContainerDied","Data":"938bbae68b4718b20a025c12524f195e2214c5e0584d347d447093091c580ad0"}
Sep 16 20:43:53.313546 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:53.313391    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="938bbae68b4718b20a025c12524f195e2214c5e0584d347d447093091c580ad0"
Sep 16 20:43:53.577154 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:53.577130    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59") on node "ip-10-0-141-26.ec2.internal"
Sep 16 20:43:53.617132 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:43:53.617103    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"csi-e04fa74e4d9acc33a95c4c8614edc5613e24db7b3712c47f712892001a69dd50\""
Sep 16 20:44:17.206252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.206218    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"]
Sep 16 20:44:17.206747 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.206577    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af0756f-e55a-49f5-838f-031bd412924e" containerName="storage-initializer"
Sep 16 20:44:17.206747 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.206589    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af0756f-e55a-49f5-838f-031bd412924e" containerName="storage-initializer"
Sep 16 20:44:17.206747 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.206655    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af0756f-e55a-49f5-838f-031bd412924e" containerName="storage-initializer"
Sep 16 20:44:17.209784 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.209767    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.212629 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.212603    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 20:44:17.212629 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.212605    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 20:44:17.212913 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.212891    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 20:44:17.212982 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.212943    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-58p6d\""
Sep 16 20:44:17.212982 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.212960    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 16 20:44:17.213609 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.213590    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:44:17.224944 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.224922    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"]
Sep 16 20:44:17.276571 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.276535    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:44:17.324578 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.324539    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlzjk\" (UniqueName: \"kubernetes.io/projected/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kube-api-access-jlzjk\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.324758 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.324585    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.324758 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.324656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-dshm\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.324873 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.324750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.324873 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.324781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-home\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.324873 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.324809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.325014 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.324876    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.325014 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.324908    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425347 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425347 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-home\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425385    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlzjk\" (UniqueName: \"kubernetes.io/projected/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kube-api-access-jlzjk\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425632 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-dshm\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425944 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425804    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425944 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425829    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-home\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.425944 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.425889    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.426160 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.426140    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.426274 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.426252    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.427843 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.427813    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-dshm\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.428087 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.428071    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:17.438568 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:17.438548    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlzjk\" (UniqueName: \"kubernetes.io/projected/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kube-api-access-jlzjk\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:19.546703 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:19.546667    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:19.552210 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:19.552180    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:19.648028 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:19.647987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:19.648195 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:19.648118    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:19.651124 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:19.651098    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") DevicePath \"csi-e04fa74e4d9acc33a95c4c8614edc5613e24db7b3712c47f712892001a69dd50\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:19.737961 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:19.737929    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/97aad42d75204328628c95717971be9e95c5ebdfa53243eec6197da38c3c3715/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:19.756556 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:19.756523    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:19.920593 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:19.920554    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:20.199475 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:20.199449    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"]
Sep 16 20:44:20.203376 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:44:20.203349    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c15e8b_9b6d_48b7_a62f_ff783a50c61c.slice/crio-2db26550b01f16aa1e8fd2dfd9e898122fdb60a44452711b2577d9f64a015c1b WatchSource:0}: Error finding container 2db26550b01f16aa1e8fd2dfd9e898122fdb60a44452711b2577d9f64a015c1b: Status 404 returned error can't find the container with id 2db26550b01f16aa1e8fd2dfd9e898122fdb60a44452711b2577d9f64a015c1b
Sep 16 20:44:20.468140 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:20.468012    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" event={"ID":"51c15e8b-9b6d-48b7-a62f-ff783a50c61c","Type":"ContainerStarted","Data":"54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f"}
Sep 16 20:44:20.468140 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:20.468078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" event={"ID":"51c15e8b-9b6d-48b7-a62f-ff783a50c61c","Type":"ContainerStarted","Data":"2db26550b01f16aa1e8fd2dfd9e898122fdb60a44452711b2577d9f64a015c1b"}
Sep 16 20:44:22.479915 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:22.479885    2574 generic.go:356] "Generic (PLEG): container finished" podID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerID="54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f" exitCode=0
Sep 16 20:44:22.480278 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:22.479962    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" event={"ID":"51c15e8b-9b6d-48b7-a62f-ff783a50c61c","Type":"ContainerDied","Data":"54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f"}
Sep 16 20:44:23.488142 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:23.488101    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" event={"ID":"51c15e8b-9b6d-48b7-a62f-ff783a50c61c","Type":"ContainerStarted","Data":"35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5"}
Sep 16 20:44:23.514522 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:23.514455    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" podStartSLOduration=4.351547188 podStartE2EDuration="6.514430164s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.162882976s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:44:23.512013441 +0000 UTC m=+3895.069358529" watchObservedRunningTime="2026-09-16 20:44:23.514430164 +0000 UTC m=+3895.071775252"
Sep 16 20:44:23.921312 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:23.921277    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:29.284383 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:29.284348    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:44:29.293013 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:29.292984    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:44:29.921314 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:29.921279    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:44:29.923062 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:29.923024    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 20:44:39.921064 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:39.921011    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 20:44:49.921776 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:49.921724    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 20:44:59.921854 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:44:59.921804    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 20:45:09.921873 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:09.921827    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 20:45:19.921659 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:19.921608    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 20:45:29.921345 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:29.921248    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 16 20:45:39.760144 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:39.760101    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:45:39.931187 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:39.931154    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:45:39.938972 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:39.938947    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:45:41.117461 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:41.117424    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:45:51.633681 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:51.633640    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:45:51.740120 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:51.740080    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:45:56.958226 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:56.958186    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:45:57.019536 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:45:57.019501    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:06.200879 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:06.200813    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw_51c15e8b-9b6d-48b7-a62f-ff783a50c61c/storage-initializer/0.log"
Sep 16 20:46:06.211502 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:06.211468    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw_51c15e8b-9b6d-48b7-a62f-ff783a50c61c/main/0.log"
Sep 16 20:46:07.197460 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:07.197426    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"]
Sep 16 20:46:07.213466 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:07.213430    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:10.293887 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:10.293841    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:22.219706 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.219665    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main" containerID="cri-o://35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5" gracePeriod=45
Sep 16 20:46:22.870537 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.870506    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:46:22.986339 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986253    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") pod \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") "
Sep 16 20:46:22.986339 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986304    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-home\") pod \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") "
Sep 16 20:46:22.986603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986342    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlzjk\" (UniqueName: \"kubernetes.io/projected/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kube-api-access-jlzjk\") pod \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") "
Sep 16 20:46:22.986603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986374    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-dshm\") pod \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") "
Sep 16 20:46:22.986603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986447    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-cabundle-cert\") pod \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") "
Sep 16 20:46:22.986603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986486    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tmp-dir\") pod \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") "
Sep 16 20:46:22.986603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986528    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-model-cache\") pod \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") "
Sep 16 20:46:22.986603 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986552    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kserve-provision-location\") pod \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") "
Sep 16 20:46:22.986905 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986607    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tls-certs\") pod \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\" (UID: \"51c15e8b-9b6d-48b7-a62f-ff783a50c61c\") "
Sep 16 20:46:22.986905 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986803    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-home" (OuterVolumeSpecName: "home") pod "51c15e8b-9b6d-48b7-a62f-ff783a50c61c" (UID: "51c15e8b-9b6d-48b7-a62f-ff783a50c61c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:22.986905 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986823    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-model-cache" (OuterVolumeSpecName: "model-cache") pod "51c15e8b-9b6d-48b7-a62f-ff783a50c61c" (UID: "51c15e8b-9b6d-48b7-a62f-ff783a50c61c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:22.986905 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986838    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51c15e8b-9b6d-48b7-a62f-ff783a50c61c" (UID: "51c15e8b-9b6d-48b7-a62f-ff783a50c61c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:46:22.987131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986931    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:46:22.987131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986951    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:46:22.987131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.986965    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:46:22.988808 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.988784    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51c15e8b-9b6d-48b7-a62f-ff783a50c61c" (UID: "51c15e8b-9b6d-48b7-a62f-ff783a50c61c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:46:22.989018 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.988989    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-dshm" (OuterVolumeSpecName: "dshm") pod "51c15e8b-9b6d-48b7-a62f-ff783a50c61c" (UID: "51c15e8b-9b6d-48b7-a62f-ff783a50c61c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:22.989149 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.989127    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kube-api-access-jlzjk" (OuterVolumeSpecName: "kube-api-access-jlzjk") pod "51c15e8b-9b6d-48b7-a62f-ff783a50c61c" (UID: "51c15e8b-9b6d-48b7-a62f-ff783a50c61c"). InnerVolumeSpecName "kube-api-access-jlzjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:46:22.991591 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.991570    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "51c15e8b-9b6d-48b7-a62f-ff783a50c61c" (UID: "51c15e8b-9b6d-48b7-a62f-ff783a50c61c"). InnerVolumeSpecName "pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 20:46:22.998313 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:22.998285    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51c15e8b-9b6d-48b7-a62f-ff783a50c61c" (UID: "51c15e8b-9b6d-48b7-a62f-ff783a50c61c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:23.016541 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.016511    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51c15e8b-9b6d-48b7-a62f-ff783a50c61c" (UID: "51c15e8b-9b6d-48b7-a62f-ff783a50c61c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:46:23.087968 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.087940    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:46:23.087968 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.087961    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:46:23.087968 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.087970    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:46:23.088204 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.088001    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") on node \"ip-10-0-141-26.ec2.internal\" "
Sep 16 20:46:23.088204 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.088014    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlzjk\" (UniqueName: \"kubernetes.io/projected/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-kube-api-access-jlzjk\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:46:23.088204 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.088024    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c15e8b-9b6d-48b7-a62f-ff783a50c61c-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:46:23.097875 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.097857    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59") on node "ip-10-0-141-26.ec2.internal"
Sep 16 20:46:23.178775 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.178737    2574 generic.go:356] "Generic (PLEG): container finished" podID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerID="35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5" exitCode=0
Sep 16 20:46:23.178916 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.178794    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"
Sep 16 20:46:23.178916 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.178793    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" event={"ID":"51c15e8b-9b6d-48b7-a62f-ff783a50c61c","Type":"ContainerDied","Data":"35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5"}
Sep 16 20:46:23.178916 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.178855    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw" event={"ID":"51c15e8b-9b6d-48b7-a62f-ff783a50c61c","Type":"ContainerDied","Data":"2db26550b01f16aa1e8fd2dfd9e898122fdb60a44452711b2577d9f64a015c1b"}
Sep 16 20:46:23.178916 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.178878    2574 scope.go:117] "RemoveContainer" containerID="35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5"
Sep 16 20:46:23.187875 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.187857    2574 scope.go:117] "RemoveContainer" containerID="54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f"
Sep 16 20:46:23.188420 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.188384    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-2ee958af-505a-401f-8ad6-ffa6f1301eba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e96c01e5ab325c59\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"csi-e04fa74e4d9acc33a95c4c8614edc5613e24db7b3712c47f712892001a69dd50\""
Sep 16 20:46:23.196630 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.196611    2574 scope.go:117] "RemoveContainer" containerID="35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5"
Sep 16 20:46:23.196935 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:46:23.196916    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5\": container with ID starting with 35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5 not found: ID does not exist" containerID="35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5"
Sep 16 20:46:23.197017 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.196943    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5"} err="failed to get container status \"35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5\": rpc error: code = NotFound desc = could not find container \"35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5\": container with ID starting with 35ee521a61cac800373fd272d6ed683d0255676bf7b7082f2e3cfc1bb275a1c5 not found: ID does not exist"
Sep 16 20:46:23.197017 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.196960    2574 scope.go:117] "RemoveContainer" containerID="54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f"
Sep 16 20:46:23.197238 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:46:23.197218    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f\": container with ID starting with 54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f not found: ID does not exist" containerID="54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f"
Sep 16 20:46:23.197280 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.197244    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f"} err="failed to get container status \"54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f\": rpc error: code = NotFound desc = could not find container \"54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f\": container with ID starting with 54737e92fa7fec34935d5a9c43f62a314d693e17e00f0d7c1fdfecd5c9b85a9f not found: ID does not exist"
Sep 16 20:46:23.197581 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.197563    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"]
Sep 16 20:46:23.208455 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:23.203447    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7f96d5cb55-2dspw"]
Sep 16 20:46:25.088965 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:25.088920    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" path="/var/lib/kubelet/pods/51c15e8b-9b6d-48b7-a62f-ff783a50c61c/volumes"
Sep 16 20:46:27.308790 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:27.308746    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:27.408829 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:27.408787    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:27.531144 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:27.531108    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:27.624568 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:27.624532    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:27.726164 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:27.726123    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:27.822505 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:27.822468    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:29.897200 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:29.897165    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:29.954130 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:29.954092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:30.770891 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:30.770853    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:32.294378 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:32.294347    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"]
Sep 16 20:46:32.298635 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:32.298611    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4c8hb"]
Sep 16 20:46:32.980210 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:32.980177    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:33.079054 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:33.079022    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:33.091496 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:33.091463    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af0756f-e55a-49f5-838f-031bd412924e" path="/var/lib/kubelet/pods/7af0756f-e55a-49f5-838f-031bd412924e/volumes"
Sep 16 20:46:33.727201 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:33.727150    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:39.523255 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:39.523198    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:39.623368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:39.623334    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:40.132305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:40.132261    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:40.185854 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:40.185819    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:47.121498 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:47.121452    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:47.220753 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:47.220718    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:53.551893 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:53.551813    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:57.069078 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.069044    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"]
Sep 16 20:46:57.069566 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.069386    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main"
Sep 16 20:46:57.069566 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.069413    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main"
Sep 16 20:46:57.069566 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.069431    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="storage-initializer"
Sep 16 20:46:57.069566 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.069437    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="storage-initializer"
Sep 16 20:46:57.069566 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.069501    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c15e8b-9b6d-48b7-a62f-ff783a50c61c" containerName="main"
Sep 16 20:46:57.071542 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.071522    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.074181 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.074154    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-dzph5\""
Sep 16 20:46:57.074432 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.074382    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 20:46:57.074583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.074474    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 20:46:57.074583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.074548    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 20:46:57.074697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.074667    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:46:57.074697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.074673    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 20:46:57.089730 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.089706    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"]
Sep 16 20:46:57.091904 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.091876    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:57.159777 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.159741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/129e090d-9336-4b08-a88b-5299c7d064f5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.159777 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.159780    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.160010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.159860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkb7h\" (UniqueName: \"kubernetes.io/projected/129e090d-9336-4b08-a88b-5299c7d064f5-kube-api-access-lkb7h\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.160010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.159942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/129e090d-9336-4b08-a88b-5299c7d064f5-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.160010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.159998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.160172 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.160016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.160172 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.160164    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.160279 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.160193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.260950 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.260915    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkb7h\" (UniqueName: \"kubernetes.io/projected/129e090d-9336-4b08-a88b-5299c7d064f5-kube-api-access-lkb7h\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.260964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/129e090d-9336-4b08-a88b-5299c7d064f5-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.260993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261057    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/129e090d-9336-4b08-a88b-5299c7d064f5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261556 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261508    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261615 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261573    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261674 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261653    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261738 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261654    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.261889 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.261868    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/129e090d-9336-4b08-a88b-5299c7d064f5-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.263389 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.263363    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.263735 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.263717    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/129e090d-9336-4b08-a88b-5299c7d064f5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.268915 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.268894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkb7h\" (UniqueName: \"kubernetes.io/projected/129e090d-9336-4b08-a88b-5299c7d064f5-kube-api-access-lkb7h\") pod \"custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.349937 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.349895    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:57.382244 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.382203    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:46:57.665596 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:57.665558    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"]
Sep 16 20:46:57.668193 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:46:57.668161    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129e090d_9336_4b08_a88b_5299c7d064f5.slice/crio-82bf3b6b1665b4982b0f354b52e368ba855c7374466c428ce9e31268a9c71870 WatchSource:0}: Error finding container 82bf3b6b1665b4982b0f354b52e368ba855c7374466c428ce9e31268a9c71870: Status 404 returned error can't find the container with id 82bf3b6b1665b4982b0f354b52e368ba855c7374466c428ce9e31268a9c71870
Sep 16 20:46:58.377782 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:58.377730    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" event={"ID":"129e090d-9336-4b08-a88b-5299c7d064f5","Type":"ContainerStarted","Data":"a12cb6b69c41821d3e6b5e8e7cc4c5fd8cc406b987c6feb8124cc01b6d790f3b"}
Sep 16 20:46:58.378148 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:58.377836    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" event={"ID":"129e090d-9336-4b08-a88b-5299c7d064f5","Type":"ContainerStarted","Data":"82bf3b6b1665b4982b0f354b52e368ba855c7374466c428ce9e31268a9c71870"}
Sep 16 20:46:59.802804 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:59.802766    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:46:59.903487 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:46:59.903452    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:47:01.394864 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:01.394828    2574 generic.go:356] "Generic (PLEG): container finished" podID="129e090d-9336-4b08-a88b-5299c7d064f5" containerID="a12cb6b69c41821d3e6b5e8e7cc4c5fd8cc406b987c6feb8124cc01b6d790f3b" exitCode=0
Sep 16 20:47:01.395260 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:01.394903    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" event={"ID":"129e090d-9336-4b08-a88b-5299c7d064f5","Type":"ContainerDied","Data":"a12cb6b69c41821d3e6b5e8e7cc4c5fd8cc406b987c6feb8124cc01b6d790f3b"}
Sep 16 20:47:02.402589 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:02.402555    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" event={"ID":"129e090d-9336-4b08-a88b-5299c7d064f5","Type":"ContainerStarted","Data":"91d2e7f67984d5d1b8e548195d6e3af0d3e8541b8ec9e1f00fd4a4bc4dac8f31"}
Sep 16 20:47:02.427116 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:02.427061    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podStartSLOduration=2.747414388 podStartE2EDuration="5.427042232s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.679627844s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:47:02.424876187 +0000 UTC m=+4053.982221294" watchObservedRunningTime="2026-09-16 20:47:02.427042232 +0000 UTC m=+4053.984387319"
Sep 16 20:47:07.383208 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:07.383163    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:47:07.383639 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:07.383249    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:47:07.384803 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:07.384775    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:47:14.290370 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:14.290327    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:47:17.383445 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:17.383373    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:47:20.319477 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:20.319427    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:47:27.383274 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:27.383225    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:47:29.540775 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:29.540741    2574 scope.go:117] "RemoveContainer" containerID="befbfae874f21c1d9e9fd610b55742c8acb7b5b588c4c539798917ae02428772"
Sep 16 20:47:37.383151 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:37.383100    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:47:47.382890 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:47.382843    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:47:57.382718 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:47:57.382673    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:48:07.382918 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:48:07.382862    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:48:17.383252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:48:17.383199    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:48:27.382874 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:48:27.382781    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:48:29.559647 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:48:29.559612    2574 scope.go:117] "RemoveContainer" containerID="30befa76d689be6d8dfd7d4daf6fd13519903a4ce59033ed63b29b94c509cf66"
Sep 16 20:48:37.383104 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:48:37.383049    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:48:47.383002 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:48:47.382946    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 16 20:48:57.393449 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:48:57.393415    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:48:57.401981 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:48:57.401952    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:49:06.467831 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:06.467793    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"]
Sep 16 20:49:06.479997 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:06.479955    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:49:06.536492 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:06.536451    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:49:21.491397 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:21.491356    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main" containerID="cri-o://91d2e7f67984d5d1b8e548195d6e3af0d3e8541b8ec9e1f00fd4a4bc4dac8f31" gracePeriod=45
Sep 16 20:49:22.188803 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.188769    2574 generic.go:356] "Generic (PLEG): container finished" podID="129e090d-9336-4b08-a88b-5299c7d064f5" containerID="91d2e7f67984d5d1b8e548195d6e3af0d3e8541b8ec9e1f00fd4a4bc4dac8f31" exitCode=0
Sep 16 20:49:22.188957 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.188846    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" event={"ID":"129e090d-9336-4b08-a88b-5299c7d064f5","Type":"ContainerDied","Data":"91d2e7f67984d5d1b8e548195d6e3af0d3e8541b8ec9e1f00fd4a4bc4dac8f31"}
Sep 16 20:49:22.242082 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.242054    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:49:22.275764 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.275684    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkb7h\" (UniqueName: \"kubernetes.io/projected/129e090d-9336-4b08-a88b-5299c7d064f5-kube-api-access-lkb7h\") pod \"129e090d-9336-4b08-a88b-5299c7d064f5\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") "
Sep 16 20:49:22.275764 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.275729    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-home\") pod \"129e090d-9336-4b08-a88b-5299c7d064f5\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") "
Sep 16 20:49:22.275956 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.275775    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-dshm\") pod \"129e090d-9336-4b08-a88b-5299c7d064f5\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") "
Sep 16 20:49:22.275956 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.275876    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-tmp-dir\") pod \"129e090d-9336-4b08-a88b-5299c7d064f5\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") "
Sep 16 20:49:22.275956 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.275938    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/129e090d-9336-4b08-a88b-5299c7d064f5-tls-certs\") pod \"129e090d-9336-4b08-a88b-5299c7d064f5\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") "
Sep 16 20:49:22.276116 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.275966    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/129e090d-9336-4b08-a88b-5299c7d064f5-cabundle-cert\") pod \"129e090d-9336-4b08-a88b-5299c7d064f5\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") "
Sep 16 20:49:22.276116 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.275998    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-model-cache\") pod \"129e090d-9336-4b08-a88b-5299c7d064f5\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") "
Sep 16 20:49:22.276116 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.276086    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-kserve-provision-location\") pod \"129e090d-9336-4b08-a88b-5299c7d064f5\" (UID: \"129e090d-9336-4b08-a88b-5299c7d064f5\") "
Sep 16 20:49:22.276429 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.276366    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "129e090d-9336-4b08-a88b-5299c7d064f5" (UID: "129e090d-9336-4b08-a88b-5299c7d064f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.276613 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.276584    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-home" (OuterVolumeSpecName: "home") pod "129e090d-9336-4b08-a88b-5299c7d064f5" (UID: "129e090d-9336-4b08-a88b-5299c7d064f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.276713 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.276649    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129e090d-9336-4b08-a88b-5299c7d064f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "129e090d-9336-4b08-a88b-5299c7d064f5" (UID: "129e090d-9336-4b08-a88b-5299c7d064f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:49:22.278505 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.278464    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129e090d-9336-4b08-a88b-5299c7d064f5-kube-api-access-lkb7h" (OuterVolumeSpecName: "kube-api-access-lkb7h") pod "129e090d-9336-4b08-a88b-5299c7d064f5" (UID: "129e090d-9336-4b08-a88b-5299c7d064f5"). InnerVolumeSpecName "kube-api-access-lkb7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:49:22.278733 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.278639    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/129e090d-9336-4b08-a88b-5299c7d064f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "129e090d-9336-4b08-a88b-5299c7d064f5" (UID: "129e090d-9336-4b08-a88b-5299c7d064f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:49:22.278902 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.278878    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-dshm" (OuterVolumeSpecName: "dshm") pod "129e090d-9336-4b08-a88b-5299c7d064f5" (UID: "129e090d-9336-4b08-a88b-5299c7d064f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.295778 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.295732    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "129e090d-9336-4b08-a88b-5299c7d064f5" (UID: "129e090d-9336-4b08-a88b-5299c7d064f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.301495 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.301452    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "129e090d-9336-4b08-a88b-5299c7d064f5" (UID: "129e090d-9336-4b08-a88b-5299c7d064f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:49:22.377368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.377333    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.377368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.377365    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkb7h\" (UniqueName: \"kubernetes.io/projected/129e090d-9336-4b08-a88b-5299c7d064f5-kube-api-access-lkb7h\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.377368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.377378    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.377644 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.377392    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.377644 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.377454    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.377644 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.377469    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/129e090d-9336-4b08-a88b-5299c7d064f5-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.377644 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.377480    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/129e090d-9336-4b08-a88b-5299c7d064f5-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:49:22.377644 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:22.377491    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/129e090d-9336-4b08-a88b-5299c7d064f5-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:49:23.195497 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:23.195466    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"
Sep 16 20:49:23.195961 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:23.195471    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7" event={"ID":"129e090d-9336-4b08-a88b-5299c7d064f5","Type":"ContainerDied","Data":"82bf3b6b1665b4982b0f354b52e368ba855c7374466c428ce9e31268a9c71870"}
Sep 16 20:49:23.195961 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:23.195603    2574 scope.go:117] "RemoveContainer" containerID="91d2e7f67984d5d1b8e548195d6e3af0d3e8541b8ec9e1f00fd4a4bc4dac8f31"
Sep 16 20:49:23.204703 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:23.204685    2574 scope.go:117] "RemoveContainer" containerID="a12cb6b69c41821d3e6b5e8e7cc4c5fd8cc406b987c6feb8124cc01b6d790f3b"
Sep 16 20:49:23.217906 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:23.217883    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"]
Sep 16 20:49:23.220926 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:23.220904    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-84f94f7db8-rp5f7"]
Sep 16 20:49:25.089191 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:25.089158    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" path="/var/lib/kubelet/pods/129e090d-9336-4b08-a88b-5299c7d064f5/volumes"
Sep 16 20:49:29.321449 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:29.321417    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:49:29.330453 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:29.330432    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:49:41.027344 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:41.027308    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:49:56.591886 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.591754    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:49:56.692267 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.692233    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:49:56.799941 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.799896    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"]
Sep 16 20:49:56.800359 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.800339    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main"
Sep 16 20:49:56.800359 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.800361    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main"
Sep 16 20:49:56.800563 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.800372    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="storage-initializer"
Sep 16 20:49:56.800563 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.800380    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="storage-initializer"
Sep 16 20:49:56.800563 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.800509    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="129e090d-9336-4b08-a88b-5299c7d064f5" containerName="main"
Sep 16 20:49:56.803732 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.803712    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.806781 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.806751    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 20:49:56.806916 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.806811    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-sjhcx\""
Sep 16 20:49:56.806916 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.806826    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 20:49:56.806916 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.806863    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 16 20:49:56.815917 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.815889    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"]
Sep 16 20:49:56.828687 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.828645    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:49:56.875526 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.875437    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-envoy\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.875526 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.875476    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-certs\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.875526 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.875503    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-podinfo\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.875772 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.875550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-token\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.875772 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.875680    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-data\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.875772 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.875724    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.875772 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.875750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.875906 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.875776    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-credential-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.875906 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.875811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzkq\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-kube-api-access-4rzkq\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.926434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.926386    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:49:56.976499 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.976461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzkq\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-kube-api-access-4rzkq\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.976671 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.976552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-envoy\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.976671 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.976573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-certs\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.976671 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.976596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-podinfo\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.976671 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.976619    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-token\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.976671 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.976663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-data\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.976935 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.976696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.976935 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.976721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.976935 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.976744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-credential-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.977118 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.977091    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-certs\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.977118 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.977108    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-credential-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.977293 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.977192    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.977452 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.977428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-data\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.977607 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.977588    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.979109 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.979087    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-envoy\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.979343 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.979327    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-podinfo\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.984104 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.984076    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-token\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:56.984247 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:56.984228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzkq\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-kube-api-access-4rzkq\") pod \"router-gateway-2-openshift-default-7f645c8cbd-5w6ps\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:57.116648 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:57.116610    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:57.392885 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:57.392845    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"]
Sep 16 20:49:57.398059 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:57.398016    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:49:57.398156 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:57.398111    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:49:57.398198 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:57.398155    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 20:49:58.384706 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:58.384665    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" event={"ID":"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2","Type":"ContainerStarted","Data":"185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e"}
Sep 16 20:49:58.385201 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:58.384758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" event={"ID":"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2","Type":"ContainerStarted","Data":"588ea6c289e1b9c70d43038a56f71c78a73b2e583ab46e18b5ffacf61082238e"}
Sep 16 20:49:58.404752 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:58.404699    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" podStartSLOduration=2.404682669 podStartE2EDuration="2.404682669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:49:58.402602953 +0000 UTC m=+4229.959948043" watchObservedRunningTime="2026-09-16 20:49:58.404682669 +0000 UTC m=+4229.962027755"
Sep 16 20:49:59.117669 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:59.117635    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:49:59.119500 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:49:59.119468    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 20:50:00.097511 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:00.087003    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:00.117203 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:00.117170    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 20:50:01.117588 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:01.117546    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 20:50:01.535694 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:01.535597    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:01.631221 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:01.631185    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:02.117493 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:02.117442    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 20:50:03.117970 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:03.117925    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 20:50:03.713203 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:03.713158    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:04.121194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:04.121156    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:50:04.121643 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:04.121457    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:50:04.122223 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:04.122202    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:50:06.977280 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:06.977243    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:07.077121 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:07.077075    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:07.672682 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:07.672647    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:09.463143 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:09.463110    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:09.530375 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:09.530339    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:13.577016 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:13.576970    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:13.647258 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:13.647220    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:14.155559 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:14.155526    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:14.226449 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:14.226394    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:17.079810 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.079778    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"]
Sep 16 20:50:17.083622 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.083593    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.087110 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.086961    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 20:50:17.087879 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.087045    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 20:50:17.088425 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.087114    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-tfs4k\""
Sep 16 20:50:17.088425 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.087192    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:50:17.103138 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.103108    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"]
Sep 16 20:50:17.151274 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.151238    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.151480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.151282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3136ea0c-cee5-4f97-ad95-25696768e7d3-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.151480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.151313    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3136ea0c-cee5-4f97-ad95-25696768e7d3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.151480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.151423    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.151480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.151453    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdwc\" (UniqueName: \"kubernetes.io/projected/3136ea0c-cee5-4f97-ad95-25696768e7d3-kube-api-access-2jdwc\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.151740 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.151483    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.151740 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.151528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.151740 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.151649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-home\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.176279 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.176235    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:17.252222 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252173    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252444 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-home\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252444 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252444 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252284    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3136ea0c-cee5-4f97-ad95-25696768e7d3-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252444 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3136ea0c-cee5-4f97-ad95-25696768e7d3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252444 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252714 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdwc\" (UniqueName: \"kubernetes.io/projected/3136ea0c-cee5-4f97-ad95-25696768e7d3-kube-api-access-2jdwc\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252793 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252718    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-home\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252793 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252693    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252793 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252760    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.252964 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.252941    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.253093 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.253071    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.253136 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.253076    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3136ea0c-cee5-4f97-ad95-25696768e7d3-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.254655 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.254637    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.255023 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.255005    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3136ea0c-cee5-4f97-ad95-25696768e7d3-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.260949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.260919    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdwc\" (UniqueName: \"kubernetes.io/projected/3136ea0c-cee5-4f97-ad95-25696768e7d3-kube-api-access-2jdwc\") pod \"router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.400859 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.400813    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:17.629353 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.629317    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:17.683897 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.683868    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"]
Sep 16 20:50:17.684585 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:50:17.684554    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3136ea0c_cee5_4f97_ad95_25696768e7d3.slice/crio-c22b753be4146df300675b56b4cc15a4bdbe0b7db91bd64e598ca1d509090add WatchSource:0}: Error finding container c22b753be4146df300675b56b4cc15a4bdbe0b7db91bd64e598ca1d509090add: Status 404 returned error can't find the container with id c22b753be4146df300675b56b4cc15a4bdbe0b7db91bd64e598ca1d509090add
Sep 16 20:50:17.692959 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:17.692929    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:18.497695 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:18.497642    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" event={"ID":"3136ea0c-cee5-4f97-ad95-25696768e7d3","Type":"ContainerStarted","Data":"f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546"}
Sep 16 20:50:18.498117 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:18.497724    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" event={"ID":"3136ea0c-cee5-4f97-ad95-25696768e7d3","Type":"ContainerStarted","Data":"c22b753be4146df300675b56b4cc15a4bdbe0b7db91bd64e598ca1d509090add"}
Sep 16 20:50:20.510066 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:20.510029    2574 generic.go:356] "Generic (PLEG): container finished" podID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerID="f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546" exitCode=0
Sep 16 20:50:20.510545 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:20.510100    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" event={"ID":"3136ea0c-cee5-4f97-ad95-25696768e7d3","Type":"ContainerDied","Data":"f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546"}
Sep 16 20:50:21.517511 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:21.517472    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" event={"ID":"3136ea0c-cee5-4f97-ad95-25696768e7d3","Type":"ContainerStarted","Data":"9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65"}
Sep 16 20:50:21.543669 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:21.543607    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podStartSLOduration=2.299912346 podStartE2EDuration="4.543589717s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.243677371s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:50:21.542185163 +0000 UTC m=+4253.099530274" watchObservedRunningTime="2026-09-16 20:50:21.543589717 +0000 UTC m=+4253.100934804"
Sep 16 20:50:22.277949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.277911    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"]
Sep 16 20:50:22.282355 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.282327    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.293940 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.293893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.294104 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.293969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.294104 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.294045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgz4l\" (UniqueName: \"kubernetes.io/projected/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kube-api-access-tgz4l\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.294104 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.294092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.294273 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.294194    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.294273 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.294244    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.294379 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.294285    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.294379 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.294323    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-home\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.297440 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.296916    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"]
Sep 16 20:50:22.395752 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.395714    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.395752 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.395756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396015 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.395783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396015 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.395802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-home\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396015 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.395950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396015 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.396012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396235 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.396078    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgz4l\" (UniqueName: \"kubernetes.io/projected/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kube-api-access-tgz4l\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396235 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.396126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396235 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.396226    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-home\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396398 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.396266    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396398 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.396318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396658 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.396635    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.396850 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.396829    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.398194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.398164    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.398862 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.398840    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.406004 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.405981    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgz4l\" (UniqueName: \"kubernetes.io/projected/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kube-api-access-tgz4l\") pod \"router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.607158 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.607122    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:22.891349 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:22.891312    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"]
Sep 16 20:50:22.893058 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:50:22.893031    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae04e9d_d328_4eec_88ff_3ae6b4ad42be.slice/crio-9a64f1a2d5fcb3924d21f5ed750d7ac431c6321e0665c56d48bee42c1c30d020 WatchSource:0}: Error finding container 9a64f1a2d5fcb3924d21f5ed750d7ac431c6321e0665c56d48bee42c1c30d020: Status 404 returned error can't find the container with id 9a64f1a2d5fcb3924d21f5ed750d7ac431c6321e0665c56d48bee42c1c30d020
Sep 16 20:50:23.531945 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:23.531526    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" event={"ID":"eae04e9d-d328-4eec-88ff-3ae6b4ad42be","Type":"ContainerStarted","Data":"6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70"}
Sep 16 20:50:23.531945 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:23.531624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" event={"ID":"eae04e9d-d328-4eec-88ff-3ae6b4ad42be","Type":"ContainerStarted","Data":"9a64f1a2d5fcb3924d21f5ed750d7ac431c6321e0665c56d48bee42c1c30d020"}
Sep 16 20:50:25.544043 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:25.544008    2574 generic.go:356] "Generic (PLEG): container finished" podID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerID="6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70" exitCode=0
Sep 16 20:50:25.544043 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:25.544026    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" event={"ID":"eae04e9d-d328-4eec-88ff-3ae6b4ad42be","Type":"ContainerDied","Data":"6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70"}
Sep 16 20:50:26.553702 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:26.553657    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" event={"ID":"eae04e9d-d328-4eec-88ff-3ae6b4ad42be","Type":"ContainerStarted","Data":"defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc"}
Sep 16 20:50:26.578921 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:26.578857    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podStartSLOduration=2.235315755 podStartE2EDuration="4.578834442s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.343518687s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:50:26.577617246 +0000 UTC m=+4258.134962358" watchObservedRunningTime="2026-09-16 20:50:26.578834442 +0000 UTC m=+4258.136179530"
Sep 16 20:50:27.401067 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:27.401022    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:27.401292 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:27.401097    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:50:27.402861 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:27.402826    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:50:29.592222 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:29.592184    2574 scope.go:117] "RemoveContainer" containerID="bf11d69cc0fdc386294d6865f701c1cbc689184cc12012a5b919d1a7733af39f"
Sep 16 20:50:30.661722 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.661688    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"]
Sep 16 20:50:30.667771 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.667741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.670535 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.670500    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Sep 16 20:50:30.670683 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.670598    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Sep 16 20:50:30.670765 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.670744    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Sep 16 20:50:30.670825 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.670509    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 20:50:30.671020 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.670998    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-mv9dj\""
Sep 16 20:50:30.671122 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.671070    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:50:30.677362 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.677330    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"]
Sep 16 20:50:30.695861 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.695825    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:30.776224 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.776187    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-home\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.776450 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.776251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-model-cache\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.776450 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.776279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slck4\" (UniqueName: \"kubernetes.io/projected/c07a6740-8404-4619-8ecd-bc1136d9d69a-kube-api-access-slck4\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.776450 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.776322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.776450 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.776431    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07a6740-8404-4619-8ecd-bc1136d9d69a-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.776718 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.776487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-dshm\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.776718 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.776585    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-tmp-dir\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.776718 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.776633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07a6740-8404-4619-8ecd-bc1136d9d69a-tls-certs\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.877807 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.877772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07a6740-8404-4619-8ecd-bc1136d9d69a-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.877848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-dshm\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.877891    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-tmp-dir\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.877913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07a6740-8404-4619-8ecd-bc1136d9d69a-tls-certs\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.877949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-home\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.878001    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-model-cache\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878284 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.878025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slck4\" (UniqueName: \"kubernetes.io/projected/c07a6740-8404-4619-8ecd-bc1136d9d69a-kube-api-access-slck4\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878284 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.878075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878538 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.878513    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878989 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.878948    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-tmp-dir\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.878989 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.878900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-home\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.879135 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.878827    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07a6740-8404-4619-8ecd-bc1136d9d69a-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.879339 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.879309    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-model-cache\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.880903 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.880839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-dshm\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.881349 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.881330    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07a6740-8404-4619-8ecd-bc1136d9d69a-tls-certs\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.891323 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.891292    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slck4\" (UniqueName: \"kubernetes.io/projected/c07a6740-8404-4619-8ecd-bc1136d9d69a-kube-api-access-slck4\") pod \"tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:30.898270 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.898225    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:30.985565 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:30.985475    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:31.272884 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:31.272669    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"]
Sep 16 20:50:31.275430 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:50:31.275386    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07a6740_8404_4619_8ecd_bc1136d9d69a.slice/crio-be03df3dcaf7c17cbbe04fae262e42c9d2c8dbd2e36e6e59a6a6205d5151f828 WatchSource:0}: Error finding container be03df3dcaf7c17cbbe04fae262e42c9d2c8dbd2e36e6e59a6a6205d5151f828: Status 404 returned error can't find the container with id be03df3dcaf7c17cbbe04fae262e42c9d2c8dbd2e36e6e59a6a6205d5151f828
Sep 16 20:50:31.588305 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:31.588248    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" event={"ID":"c07a6740-8404-4619-8ecd-bc1136d9d69a","Type":"ContainerStarted","Data":"e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89"}
Sep 16 20:50:31.588542 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:31.588374    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" event={"ID":"c07a6740-8404-4619-8ecd-bc1136d9d69a","Type":"ContainerStarted","Data":"be03df3dcaf7c17cbbe04fae262e42c9d2c8dbd2e36e6e59a6a6205d5151f828"}
Sep 16 20:50:32.608187 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:32.608149    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:32.608711 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:32.608220    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:50:32.609395 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:32.609367    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:50:34.616570 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:34.616521    2574 generic.go:356] "Generic (PLEG): container finished" podID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerID="e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89" exitCode=0
Sep 16 20:50:34.617636 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:34.616645    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" event={"ID":"c07a6740-8404-4619-8ecd-bc1136d9d69a","Type":"ContainerDied","Data":"e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89"}
Sep 16 20:50:35.626342 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:35.626304    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" event={"ID":"c07a6740-8404-4619-8ecd-bc1136d9d69a","Type":"ContainerStarted","Data":"f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3"}
Sep 16 20:50:35.652447 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:35.652368    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podStartSLOduration=2.816452436 podStartE2EDuration="5.652344981s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.835892545s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:50:35.648509367 +0000 UTC m=+4267.205854478" watchObservedRunningTime="2026-09-16 20:50:35.652344981 +0000 UTC m=+4267.209690069"
Sep 16 20:50:37.401761 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:37.401709    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:50:40.986380 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:40.986333    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:40.986883 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:40.986463    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:50:40.988501 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:40.988461    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:50:42.608092 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:42.608042    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:50:47.402079 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:47.402026    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:50:49.213825 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:49.213778    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:50.987027 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:50.986965    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:50:52.485561 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:52.485519    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:52.587797 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:52.587749    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:50:52.607981 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:52.607933    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:50:57.402050 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:50:57.401997    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:51:00.986329 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:00.986281    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:51:02.608110 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:02.608052    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:51:07.401319 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:07.401266    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:51:10.986224 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:10.986176    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:51:11.752609 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:11.752570    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:51:11.811924 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:11.811876    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:51:12.608368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:12.608315    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:51:17.401513 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:17.401460    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:51:20.986949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:20.986903    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:51:22.608244 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:22.608194    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:51:27.401374 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:27.401320    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:51:30.986617 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:30.986564    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:51:32.608544 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:32.608496    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:51:37.402114 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:37.402059    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:51:40.987030 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:40.986944    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:51:42.607798 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:42.607742    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:51:47.402107 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:47.402055    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:51:50.986129 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:50.986070    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:51:52.607652 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:52.607602    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:51:57.401562 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:51:57.401512    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:52:00.986761 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:00.986699    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:52:02.608166 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:02.608114    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:52:07.402178 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:07.402133    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:52:10.986472 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:10.986429    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:52:12.608137 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:12.608081    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:52:17.402135 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:17.402089    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:52:20.986901 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:20.986859    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:52:22.607759 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:22.607713    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:52:27.401552 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:27.401510    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:52:30.986684 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:30.986635    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:52:32.607784 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:32.607728    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:52:37.402010 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:37.401964    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:52:40.986558 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:40.986505    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:52:42.607745 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:42.607694    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:52:47.401931 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:47.401867    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:52:50.986808 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:50.986764    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:52:52.608275 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:52.608217    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:52:57.401845 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:52:57.401796    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:53:00.986366 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:00.986317    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:53:02.608122 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:02.608077    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:53:07.401681 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:07.401628    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:53:10.986830 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:10.986767    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:53:12.608156 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:12.608111    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:53:17.411522 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:17.411477    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:53:17.419546 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:17.419515    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:53:20.986892 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:20.986845    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:53:22.608480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:22.608429    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 20:53:30.986543 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:30.986495    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 20:53:32.306463 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:32.306429    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"]
Sep 16 20:53:32.314854 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:32.314828    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"]
Sep 16 20:53:32.327085 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:32.326990    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:53:32.378086 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:32.378045    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:53:40.996763 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:40.996716    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:53:41.005212 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:41.005178    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:53:41.867598 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:41.867550    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:53:41.921150 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:41.921111    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:53:47.339436 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:47.339359    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" containerID="cri-o://9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65" gracePeriod=45
Sep 16 20:53:47.347840 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:47.347803    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main" containerID="cri-o://defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc" gracePeriod=45
Sep 16 20:53:47.412841 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:47.412789    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 16 20:53:48.296886 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.296857    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:53:48.427999 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.427957    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3136ea0c-cee5-4f97-ad95-25696768e7d3-cabundle-cert\") pod \"3136ea0c-cee5-4f97-ad95-25696768e7d3\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") "
Sep 16 20:53:48.427999 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428002    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-home\") pod \"3136ea0c-cee5-4f97-ad95-25696768e7d3\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") "
Sep 16 20:53:48.428583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428047    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jdwc\" (UniqueName: \"kubernetes.io/projected/3136ea0c-cee5-4f97-ad95-25696768e7d3-kube-api-access-2jdwc\") pod \"3136ea0c-cee5-4f97-ad95-25696768e7d3\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") "
Sep 16 20:53:48.428583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428107    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-model-cache\") pod \"3136ea0c-cee5-4f97-ad95-25696768e7d3\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") "
Sep 16 20:53:48.428583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428145    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-dshm\") pod \"3136ea0c-cee5-4f97-ad95-25696768e7d3\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") "
Sep 16 20:53:48.428583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428198    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3136ea0c-cee5-4f97-ad95-25696768e7d3-tls-certs\") pod \"3136ea0c-cee5-4f97-ad95-25696768e7d3\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") "
Sep 16 20:53:48.428583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428262    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-kserve-provision-location\") pod \"3136ea0c-cee5-4f97-ad95-25696768e7d3\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") "
Sep 16 20:53:48.428583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428298    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-tmp-dir\") pod \"3136ea0c-cee5-4f97-ad95-25696768e7d3\" (UID: \"3136ea0c-cee5-4f97-ad95-25696768e7d3\") "
Sep 16 20:53:48.428583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428368    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3136ea0c-cee5-4f97-ad95-25696768e7d3" (UID: "3136ea0c-cee5-4f97-ad95-25696768e7d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.428583 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428387    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3136ea0c-cee5-4f97-ad95-25696768e7d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3136ea0c-cee5-4f97-ad95-25696768e7d3" (UID: "3136ea0c-cee5-4f97-ad95-25696768e7d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:53:48.429030 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428658    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-home" (OuterVolumeSpecName: "home") pod "3136ea0c-cee5-4f97-ad95-25696768e7d3" (UID: "3136ea0c-cee5-4f97-ad95-25696768e7d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.429030 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428722    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3136ea0c-cee5-4f97-ad95-25696768e7d3-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.429030 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.428746    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.430761 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.430729    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3136ea0c-cee5-4f97-ad95-25696768e7d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3136ea0c-cee5-4f97-ad95-25696768e7d3" (UID: "3136ea0c-cee5-4f97-ad95-25696768e7d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:53:48.431285 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.431257    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3136ea0c-cee5-4f97-ad95-25696768e7d3-kube-api-access-2jdwc" (OuterVolumeSpecName: "kube-api-access-2jdwc") pod "3136ea0c-cee5-4f97-ad95-25696768e7d3" (UID: "3136ea0c-cee5-4f97-ad95-25696768e7d3"). InnerVolumeSpecName "kube-api-access-2jdwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:53:48.431396 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.431260    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-dshm" (OuterVolumeSpecName: "dshm") pod "3136ea0c-cee5-4f97-ad95-25696768e7d3" (UID: "3136ea0c-cee5-4f97-ad95-25696768e7d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.440788 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.440762    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3136ea0c-cee5-4f97-ad95-25696768e7d3" (UID: "3136ea0c-cee5-4f97-ad95-25696768e7d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.464916 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.464858    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3136ea0c-cee5-4f97-ad95-25696768e7d3" (UID: "3136ea0c-cee5-4f97-ad95-25696768e7d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:53:48.529780 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.529744    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3136ea0c-cee5-4f97-ad95-25696768e7d3-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.529780 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.529779    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.529960 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.529789    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.529960 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.529800    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.529960 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.529808    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdwc\" (UniqueName: \"kubernetes.io/projected/3136ea0c-cee5-4f97-ad95-25696768e7d3-kube-api-access-2jdwc\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.529960 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.529816    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3136ea0c-cee5-4f97-ad95-25696768e7d3-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:53:48.820509 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.820414    2574 generic.go:356] "Generic (PLEG): container finished" podID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerID="9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65" exitCode=0
Sep 16 20:53:48.820509 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.820476    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"
Sep 16 20:53:48.820509 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.820501    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" event={"ID":"3136ea0c-cee5-4f97-ad95-25696768e7d3","Type":"ContainerDied","Data":"9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65"}
Sep 16 20:53:48.820738 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.820562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779" event={"ID":"3136ea0c-cee5-4f97-ad95-25696768e7d3","Type":"ContainerDied","Data":"c22b753be4146df300675b56b4cc15a4bdbe0b7db91bd64e598ca1d509090add"}
Sep 16 20:53:48.820738 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.820593    2574 scope.go:117] "RemoveContainer" containerID="9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65"
Sep 16 20:53:48.831338 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.831317    2574 scope.go:117] "RemoveContainer" containerID="f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546"
Sep 16 20:53:48.841257 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.841235    2574 scope.go:117] "RemoveContainer" containerID="9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65"
Sep 16 20:53:48.842162 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:53:48.842128    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65\": container with ID starting with 9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65 not found: ID does not exist" containerID="9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65"
Sep 16 20:53:48.842268 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.842168    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65"} err="failed to get container status \"9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65\": rpc error: code = NotFound desc = could not find container \"9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65\": container with ID starting with 9a09049d468e2efa841a5ff9f756a76c1bab8124e1add358c91523e1da95bf65 not found: ID does not exist"
Sep 16 20:53:48.842268 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.842193    2574 scope.go:117] "RemoveContainer" containerID="f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546"
Sep 16 20:53:48.842617 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:53:48.842587    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546\": container with ID starting with f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546 not found: ID does not exist" containerID="f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546"
Sep 16 20:53:48.842736 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.842626    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546"} err="failed to get container status \"f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546\": rpc error: code = NotFound desc = could not find container \"f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546\": container with ID starting with f4f393e1dfd5ab9fc5bb978218002b3efa764f9e33976722c45326ecaeed7546 not found: ID does not exist"
Sep 16 20:53:48.845060 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.845039    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"]
Sep 16 20:53:48.847848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:48.847825    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5cf8fbb99d-8d779"]
Sep 16 20:53:49.090285 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:49.090252    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" path="/var/lib/kubelet/pods/3136ea0c-cee5-4f97-ad95-25696768e7d3/volumes"
Sep 16 20:53:50.897021 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:50.896587    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:53:50.897021 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:50.896776    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"]
Sep 16 20:53:50.960467 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:50.960426    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:53:52.449608 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:53:52.449569    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:54:05.922153 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:05.922091    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main" containerID="cri-o://f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3" gracePeriod=45
Sep 16 20:54:06.684541 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.684480    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:54:06.797473 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.797437    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-dshm\") pod \"c07a6740-8404-4619-8ecd-bc1136d9d69a\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") "
Sep 16 20:54:06.797646 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.797516    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07a6740-8404-4619-8ecd-bc1136d9d69a-tls-certs\") pod \"c07a6740-8404-4619-8ecd-bc1136d9d69a\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") "
Sep 16 20:54:06.797646 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.797569    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07a6740-8404-4619-8ecd-bc1136d9d69a-cabundle-cert\") pod \"c07a6740-8404-4619-8ecd-bc1136d9d69a\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") "
Sep 16 20:54:06.797646 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.797597    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slck4\" (UniqueName: \"kubernetes.io/projected/c07a6740-8404-4619-8ecd-bc1136d9d69a-kube-api-access-slck4\") pod \"c07a6740-8404-4619-8ecd-bc1136d9d69a\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") "
Sep 16 20:54:06.797646 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.797627    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-kserve-provision-location\") pod \"c07a6740-8404-4619-8ecd-bc1136d9d69a\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") "
Sep 16 20:54:06.797646 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.797645    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-home\") pod \"c07a6740-8404-4619-8ecd-bc1136d9d69a\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") "
Sep 16 20:54:06.797921 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.797667    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-tmp-dir\") pod \"c07a6740-8404-4619-8ecd-bc1136d9d69a\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") "
Sep 16 20:54:06.797921 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.797706    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-model-cache\") pod \"c07a6740-8404-4619-8ecd-bc1136d9d69a\" (UID: \"c07a6740-8404-4619-8ecd-bc1136d9d69a\") "
Sep 16 20:54:06.798027 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.797922    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07a6740-8404-4619-8ecd-bc1136d9d69a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c07a6740-8404-4619-8ecd-bc1136d9d69a" (UID: "c07a6740-8404-4619-8ecd-bc1136d9d69a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:54:06.798237 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.798211    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c07a6740-8404-4619-8ecd-bc1136d9d69a" (UID: "c07a6740-8404-4619-8ecd-bc1136d9d69a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.798339 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.798260    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-home" (OuterVolumeSpecName: "home") pod "c07a6740-8404-4619-8ecd-bc1136d9d69a" (UID: "c07a6740-8404-4619-8ecd-bc1136d9d69a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.799946 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.799917    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07a6740-8404-4619-8ecd-bc1136d9d69a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c07a6740-8404-4619-8ecd-bc1136d9d69a" (UID: "c07a6740-8404-4619-8ecd-bc1136d9d69a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:54:06.800157 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.800138    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-dshm" (OuterVolumeSpecName: "dshm") pod "c07a6740-8404-4619-8ecd-bc1136d9d69a" (UID: "c07a6740-8404-4619-8ecd-bc1136d9d69a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.800226 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.800154    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07a6740-8404-4619-8ecd-bc1136d9d69a-kube-api-access-slck4" (OuterVolumeSpecName: "kube-api-access-slck4") pod "c07a6740-8404-4619-8ecd-bc1136d9d69a" (UID: "c07a6740-8404-4619-8ecd-bc1136d9d69a"). InnerVolumeSpecName "kube-api-access-slck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:54:06.818005 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.817971    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c07a6740-8404-4619-8ecd-bc1136d9d69a" (UID: "c07a6740-8404-4619-8ecd-bc1136d9d69a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.825255 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.825231    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c07a6740-8404-4619-8ecd-bc1136d9d69a" (UID: "c07a6740-8404-4619-8ecd-bc1136d9d69a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:06.899277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.899237    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.899277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.899271    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.899277 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.899280    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c07a6740-8404-4619-8ecd-bc1136d9d69a-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.899532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.899290    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c07a6740-8404-4619-8ecd-bc1136d9d69a-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.899532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.899299    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slck4\" (UniqueName: \"kubernetes.io/projected/c07a6740-8404-4619-8ecd-bc1136d9d69a-kube-api-access-slck4\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.899532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.899309    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.899532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.899317    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.899532 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.899326    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07a6740-8404-4619-8ecd-bc1136d9d69a-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:06.927829 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.927797    2574 generic.go:356] "Generic (PLEG): container finished" podID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerID="f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3" exitCode=0
Sep 16 20:54:06.928246 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.927850    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"
Sep 16 20:54:06.928246 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.927884    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" event={"ID":"c07a6740-8404-4619-8ecd-bc1136d9d69a","Type":"ContainerDied","Data":"f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3"}
Sep 16 20:54:06.928246 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.927946    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6" event={"ID":"c07a6740-8404-4619-8ecd-bc1136d9d69a","Type":"ContainerDied","Data":"be03df3dcaf7c17cbbe04fae262e42c9d2c8dbd2e36e6e59a6a6205d5151f828"}
Sep 16 20:54:06.928246 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.927987    2574 scope.go:117] "RemoveContainer" containerID="f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3"
Sep 16 20:54:06.937952 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.937935    2574 scope.go:117] "RemoveContainer" containerID="e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89"
Sep 16 20:54:06.946167 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.946149    2574 scope.go:117] "RemoveContainer" containerID="f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3"
Sep 16 20:54:06.946446 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:54:06.946423    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3\": container with ID starting with f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3 not found: ID does not exist" containerID="f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3"
Sep 16 20:54:06.946559 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.946453    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3"} err="failed to get container status \"f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3\": rpc error: code = NotFound desc = could not find container \"f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3\": container with ID starting with f56ffd057b4fbfaab586c24586e17fcf98666d4182ebf0fe726bd76d263b26b3 not found: ID does not exist"
Sep 16 20:54:06.946559 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.946477    2574 scope.go:117] "RemoveContainer" containerID="e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89"
Sep 16 20:54:06.946739 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:54:06.946721    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89\": container with ID starting with e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89 not found: ID does not exist" containerID="e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89"
Sep 16 20:54:06.946779 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.946747    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89"} err="failed to get container status \"e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89\": rpc error: code = NotFound desc = could not find container \"e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89\": container with ID starting with e07e81a55120725df9668319bc084517054e8cb56436beb4efa8f19aec11dd89 not found: ID does not exist"
Sep 16 20:54:06.950810 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.950787    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"]
Sep 16 20:54:06.955985 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:06.955964    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-57f4dcf86f-4q9c6"]
Sep 16 20:54:07.089138 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:07.089103    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" path="/var/lib/kubelet/pods/c07a6740-8404-4619-8ecd-bc1136d9d69a/volumes"
Sep 16 20:54:29.355648 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:29.355565    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:54:29.372495 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:29.372466    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:54:32.590921 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.590890    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:54:32.626690 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.626617    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgz4l\" (UniqueName: \"kubernetes.io/projected/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kube-api-access-tgz4l\") pod \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") "
Sep 16 20:54:32.626850 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.626688    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-dshm\") pod \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") "
Sep 16 20:54:32.626850 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.626716    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-model-cache\") pod \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") "
Sep 16 20:54:32.626850 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.626821    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-home\") pod \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") "
Sep 16 20:54:32.627042 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.626872    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tmp-dir\") pod \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") "
Sep 16 20:54:32.627042 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.626904    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tls-certs\") pod \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") "
Sep 16 20:54:32.627042 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.626943    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-cabundle-cert\") pod \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") "
Sep 16 20:54:32.627042 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.626972    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kserve-provision-location\") pod \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\" (UID: \"eae04e9d-d328-4eec-88ff-3ae6b4ad42be\") "
Sep 16 20:54:32.627042 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.626988    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-model-cache" (OuterVolumeSpecName: "model-cache") pod "eae04e9d-d328-4eec-88ff-3ae6b4ad42be" (UID: "eae04e9d-d328-4eec-88ff-3ae6b4ad42be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:32.627316 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.627293    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eae04e9d-d328-4eec-88ff-3ae6b4ad42be" (UID: "eae04e9d-d328-4eec-88ff-3ae6b4ad42be"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:54:32.627380 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.627307    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:32.627490 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.627462    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-home" (OuterVolumeSpecName: "home") pod "eae04e9d-d328-4eec-88ff-3ae6b4ad42be" (UID: "eae04e9d-d328-4eec-88ff-3ae6b4ad42be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:32.629153 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.629127    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-dshm" (OuterVolumeSpecName: "dshm") pod "eae04e9d-d328-4eec-88ff-3ae6b4ad42be" (UID: "eae04e9d-d328-4eec-88ff-3ae6b4ad42be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:32.629446 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.629396    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kube-api-access-tgz4l" (OuterVolumeSpecName: "kube-api-access-tgz4l") pod "eae04e9d-d328-4eec-88ff-3ae6b4ad42be" (UID: "eae04e9d-d328-4eec-88ff-3ae6b4ad42be"). InnerVolumeSpecName "kube-api-access-tgz4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:54:32.629865 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.629839    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eae04e9d-d328-4eec-88ff-3ae6b4ad42be" (UID: "eae04e9d-d328-4eec-88ff-3ae6b4ad42be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:54:32.647115 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.647081    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eae04e9d-d328-4eec-88ff-3ae6b4ad42be" (UID: "eae04e9d-d328-4eec-88ff-3ae6b4ad42be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:32.656003 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.655977    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eae04e9d-d328-4eec-88ff-3ae6b4ad42be" (UID: "eae04e9d-d328-4eec-88ff-3ae6b4ad42be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:32.728742 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.728708    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:32.728742 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.728737    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:32.728742 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.728745    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:32.728962 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.728755    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:32.728962 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.728765    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:32.728962 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.728773    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:32.728962 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:32.728781    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgz4l\" (UniqueName: \"kubernetes.io/projected/eae04e9d-d328-4eec-88ff-3ae6b4ad42be-kube-api-access-tgz4l\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:33.070356 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.070256    2574 generic.go:356] "Generic (PLEG): container finished" podID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerID="defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc" exitCode=137
Sep 16 20:54:33.070356 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.070312    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"
Sep 16 20:54:33.070356 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.070342    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" event={"ID":"eae04e9d-d328-4eec-88ff-3ae6b4ad42be","Type":"ContainerDied","Data":"defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc"}
Sep 16 20:54:33.070602 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.070435    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4" event={"ID":"eae04e9d-d328-4eec-88ff-3ae6b4ad42be","Type":"ContainerDied","Data":"9a64f1a2d5fcb3924d21f5ed750d7ac431c6321e0665c56d48bee42c1c30d020"}
Sep 16 20:54:33.070602 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.070462    2574 scope.go:117] "RemoveContainer" containerID="defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc"
Sep 16 20:54:33.080283 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.080268    2574 scope.go:117] "RemoveContainer" containerID="6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70"
Sep 16 20:54:33.089575 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.089559    2574 scope.go:117] "RemoveContainer" containerID="defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc"
Sep 16 20:54:33.089826 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:54:33.089805    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc\": container with ID starting with defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc not found: ID does not exist" containerID="defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc"
Sep 16 20:54:33.089878 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.089836    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc"} err="failed to get container status \"defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc\": rpc error: code = NotFound desc = could not find container \"defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc\": container with ID starting with defa098968068ba905ed29af1b12f36efb5d8d8808828ba7274807ae6d2b19cc not found: ID does not exist"
Sep 16 20:54:33.089878 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.089854    2574 scope.go:117] "RemoveContainer" containerID="6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70"
Sep 16 20:54:33.090139 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:54:33.090118    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70\": container with ID starting with 6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70 not found: ID does not exist" containerID="6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70"
Sep 16 20:54:33.090239 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.090142    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70"} err="failed to get container status \"6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70\": rpc error: code = NotFound desc = could not find container \"6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70\": container with ID starting with 6979a2eae57d758bd5f4e0c9458037f75f65205c07b66775d30cbeefa7a52b70 not found: ID does not exist"
Sep 16 20:54:33.093292 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.093270    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"]
Sep 16 20:54:33.099607 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:33.099583    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6859b8f7cf-wqvg4"]
Sep 16 20:54:35.088668 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:35.088632    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" path="/var/lib/kubelet/pods/eae04e9d-d328-4eec-88ff-3ae6b4ad42be/volumes"
Sep 16 20:54:37.646519 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:37.646480    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:54:37.743004 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:37.742967    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:54:41.056256 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:41.056220    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:54:41.151505 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:41.151467    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:54:42.602623 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:42.602586    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"]
Sep 16 20:54:42.603076 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:42.602790    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerName="istio-proxy" containerID="cri-o://185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e" gracePeriod=30
Sep 16 20:54:42.697489 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:42.697452    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:54:47.394803 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.394756    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:54:47.855939 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.855912    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:54:47.962949 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.962913    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istiod-ca-cert\") pod \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") "
Sep 16 20:54:47.963131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.962969    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-token\") pod \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") "
Sep 16 20:54:47.963131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963009    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-certs\") pod \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") "
Sep 16 20:54:47.963131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963037    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-credential-socket\") pod \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") "
Sep 16 20:54:47.963131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963067    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-envoy\") pod \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") "
Sep 16 20:54:47.963131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963088    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-data\") pod \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") "
Sep 16 20:54:47.963369 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963167    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-podinfo\") pod \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") "
Sep 16 20:54:47.963369 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963214    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-socket\") pod \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") "
Sep 16 20:54:47.963369 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963239    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rzkq\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-kube-api-access-4rzkq\") pod \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\" (UID: \"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2\") "
Sep 16 20:54:47.963556 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963384    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" (UID: "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:54:47.963621 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963554    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" (UID: "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:47.963705 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963673    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:47.963705 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963700    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istiod-ca-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:47.963828 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963703    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" (UID: "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:47.963828 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963379    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" (UID: "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:47.963828 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.963788    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" (UID: "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:47.965261 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.965232    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" (UID: "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:54:47.965450 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.965423    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-kube-api-access-4rzkq" (OuterVolumeSpecName: "kube-api-access-4rzkq") pod "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" (UID: "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2"). InnerVolumeSpecName "kube-api-access-4rzkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:54:47.965642 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.965623    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" (UID: "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 20:54:47.965693 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:47.965669    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" (UID: "2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:54:48.064652 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.064614    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-token\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:48.064652 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.064648    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-workload-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:48.064868 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.064661    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-credential-socket\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:48.064868 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.064675    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-envoy\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:48.064868 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.064684    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-data\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:48.064868 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.064692    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-istio-podinfo\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:48.064868 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.064701    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rzkq\" (UniqueName: \"kubernetes.io/projected/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2-kube-api-access-4rzkq\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:54:48.149667 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.149631    2574 generic.go:356] "Generic (PLEG): container finished" podID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerID="185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e" exitCode=0
Sep 16 20:54:48.149861 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.149678    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"
Sep 16 20:54:48.149861 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.149718    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" event={"ID":"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2","Type":"ContainerDied","Data":"185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e"}
Sep 16 20:54:48.149861 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.149769    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps" event={"ID":"2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2","Type":"ContainerDied","Data":"588ea6c289e1b9c70d43038a56f71c78a73b2e583ab46e18b5ffacf61082238e"}
Sep 16 20:54:48.149861 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.149785    2574 scope.go:117] "RemoveContainer" containerID="185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e"
Sep 16 20:54:48.159537 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.159519    2574 scope.go:117] "RemoveContainer" containerID="185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e"
Sep 16 20:54:48.159827 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:54:48.159805    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e\": container with ID starting with 185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e not found: ID does not exist" containerID="185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e"
Sep 16 20:54:48.159880 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.159836    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e"} err="failed to get container status \"185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e\": rpc error: code = NotFound desc = could not find container \"185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e\": container with ID starting with 185ccd6e21862d8e73779bb344359ad5a69ddc7c62086334f509c64dbf824c3e not found: ID does not exist"
Sep 16 20:54:48.173441 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.173414    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"]
Sep 16 20:54:48.177612 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:48.177591    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-5w6ps"]
Sep 16 20:54:49.088384 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:49.088341    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" path="/var/lib/kubelet/pods/2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2/volumes"
Sep 16 20:54:53.115970 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:53.115937    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:54:53.174577 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:53.174535    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:54:58.599778 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:54:58.599741    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:02.223425 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:02.223369    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:04.275777 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:04.275708    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:05.595457 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:05.595393    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:11.619072 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:11.619031    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:25.560623 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:25.560581    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:26.800267 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:26.800227    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:26.900645 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:26.900613    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:31.593679 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.593640    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"]
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.593972    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="storage-initializer"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.593983    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="storage-initializer"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594000    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerName="istio-proxy"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594006    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerName="istio-proxy"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594013    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="storage-initializer"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594019    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="storage-initializer"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594025    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="storage-initializer"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594031    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="storage-initializer"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594043    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594048    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594054    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594059    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main"
Sep 16 20:55:31.594063 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594067    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main"
Sep 16 20:55:31.594591 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594072    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main"
Sep 16 20:55:31.594591 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594131    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c07a6740-8404-4619-8ecd-bc1136d9d69a" containerName="main"
Sep 16 20:55:31.594591 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594142    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eae04e9d-d328-4eec-88ff-3ae6b4ad42be" containerName="main"
Sep 16 20:55:31.594591 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594149    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d36e9f5-fa8f-40e8-bef7-631cd6b91ed2" containerName="istio-proxy"
Sep 16 20:55:31.594591 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.594154    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3136ea0c-cee5-4f97-ad95-25696768e7d3" containerName="main"
Sep 16 20:55:31.598748 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.598727    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.601061 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.601033    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:55:31.601233 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.601214    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 20:55:31.601375 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.601356    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 20:55:31.601492 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.601471    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 20:55:31.601595 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.601580    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-2t9hg\""
Sep 16 20:55:31.601697 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.601680    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 20:55:31.609036 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.609009    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"]
Sep 16 20:55:31.619216 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.619192    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:31.637919 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.637891    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-tmp-dir\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.638070 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.637936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-model-cache\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.638070 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.637971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-dshm\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.638070 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.638008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-home\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.638070 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.638035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e72ef58-125c-4933-bc7d-c03d3647de23-tls-certs\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.638070 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.638060    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glv4j\" (UniqueName: \"kubernetes.io/projected/2e72ef58-125c-4933-bc7d-c03d3647de23-kube-api-access-glv4j\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.638285 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.638097    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.638285 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.638172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e72ef58-125c-4933-bc7d-c03d3647de23-cabundle-cert\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739329 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e72ef58-125c-4933-bc7d-c03d3647de23-cabundle-cert\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739553 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-tmp-dir\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739553 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-model-cache\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739553 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739443    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-dshm\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739553 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739483    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-home\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739553 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e72ef58-125c-4933-bc7d-c03d3647de23-tls-certs\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739553 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glv4j\" (UniqueName: \"kubernetes.io/projected/2e72ef58-125c-4933-bc7d-c03d3647de23-kube-api-access-glv4j\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739837 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739897 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739837    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-tmp-dir\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.739975 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.739952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-model-cache\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.740071 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.740047    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-home\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.740134 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.740092    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.740230 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.740201    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e72ef58-125c-4933-bc7d-c03d3647de23-cabundle-cert\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.741906 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.741879    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-dshm\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.742098 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.742079    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e72ef58-125c-4933-bc7d-c03d3647de23-tls-certs\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.747833 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.747811    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glv4j\" (UniqueName: \"kubernetes.io/projected/2e72ef58-125c-4933-bc7d-c03d3647de23-kube-api-access-glv4j\") pod \"prestop-hook-test-kserve-5c5b674df-qkjps\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:31.821499 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.821469    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:55:31.910146 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:31.910064    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:32.189486 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:32.189456    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"]
Sep 16 20:55:32.190505 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:55:32.190478    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e72ef58_125c_4933_bc7d_c03d3647de23.slice/crio-7ae7cdfa630b1c825baa3916ff3fda92a9fdb04c358e9fb26fd2f45f546b5f90 WatchSource:0}: Error finding container 7ae7cdfa630b1c825baa3916ff3fda92a9fdb04c358e9fb26fd2f45f546b5f90: Status 404 returned error can't find the container with id 7ae7cdfa630b1c825baa3916ff3fda92a9fdb04c358e9fb26fd2f45f546b5f90
Sep 16 20:55:32.382363 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:32.382329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" event={"ID":"2e72ef58-125c-4933-bc7d-c03d3647de23","Type":"ContainerStarted","Data":"8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91"}
Sep 16 20:55:32.382559 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:32.382390    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" event={"ID":"2e72ef58-125c-4933-bc7d-c03d3647de23","Type":"ContainerStarted","Data":"7ae7cdfa630b1c825baa3916ff3fda92a9fdb04c358e9fb26fd2f45f546b5f90"}
Sep 16 20:55:35.399243 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:35.399207    2574 generic.go:356] "Generic (PLEG): container finished" podID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerID="8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91" exitCode=0
Sep 16 20:55:35.399643 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:35.399278    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" event={"ID":"2e72ef58-125c-4933-bc7d-c03d3647de23","Type":"ContainerDied","Data":"8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91"}
Sep 16 20:55:36.407467 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:36.407427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" event={"ID":"2e72ef58-125c-4933-bc7d-c03d3647de23","Type":"ContainerStarted","Data":"a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda"}
Sep 16 20:55:36.430096 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:36.430031    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podStartSLOduration=3.182112867 podStartE2EDuration="5.430009795s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.247896928s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:55:36.428658845 +0000 UTC m=+4567.986003981" watchObservedRunningTime="2026-09-16 20:55:36.430009795 +0000 UTC m=+4567.987354885"
Sep 16 20:55:36.910194 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:36.910154    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:41.910643 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:41.910588    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:55:41.912476 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:41.912444    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:55:51.911586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:55:51.911498    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:56:01.910573 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:56:01.910534    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:56:11.910919 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:56:11.910869    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:56:17.575733 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:56:17.575688    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:56:21.911393 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:56:21.911333    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:56:31.911253 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:56:31.911201    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:56:41.911366 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:56:41.911318    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:56:51.911611 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:56:51.911562    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:57:01.910839 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:01.910787    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:57:07.254041 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:07.253992    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:11.910789 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:11.910741    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:57:21.911506 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:21.911384    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 16 20:57:27.317597 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:27.317552    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:27.416839 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:27.416799    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:27.516609 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:27.516577    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:31.920480 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:31.920440    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:57:31.928558 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:31.928520    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:57:32.160328 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:32.160294    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"]
Sep 16 20:57:33.984089 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:33.984048    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:37.706614 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:37.706581    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:37.939286 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:37.939246    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:39.882016 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:39.881979    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:39.982248 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:39.982213    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:42.818894 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:42.818854    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:57:48.094570 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.094524    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main" containerID="cri-o://a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda" gracePeriod=45
Sep 16 20:57:48.843301 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.843272    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:57:48.871284 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.871245    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glv4j\" (UniqueName: \"kubernetes.io/projected/2e72ef58-125c-4933-bc7d-c03d3647de23-kube-api-access-glv4j\") pod \"2e72ef58-125c-4933-bc7d-c03d3647de23\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") "
Sep 16 20:57:48.871284 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.871294    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e72ef58-125c-4933-bc7d-c03d3647de23-cabundle-cert\") pod \"2e72ef58-125c-4933-bc7d-c03d3647de23\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") "
Sep 16 20:57:48.871573 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.871380    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e72ef58-125c-4933-bc7d-c03d3647de23-tls-certs\") pod \"2e72ef58-125c-4933-bc7d-c03d3647de23\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") "
Sep 16 20:57:48.871573 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.871450    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-kserve-provision-location\") pod \"2e72ef58-125c-4933-bc7d-c03d3647de23\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") "
Sep 16 20:57:48.871573 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.871488    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-home\") pod \"2e72ef58-125c-4933-bc7d-c03d3647de23\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") "
Sep 16 20:57:48.871573 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.871523    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-model-cache\") pod \"2e72ef58-125c-4933-bc7d-c03d3647de23\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") "
Sep 16 20:57:48.871573 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.871567    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-tmp-dir\") pod \"2e72ef58-125c-4933-bc7d-c03d3647de23\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") "
Sep 16 20:57:48.871838 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.871590    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-dshm\") pod \"2e72ef58-125c-4933-bc7d-c03d3647de23\" (UID: \"2e72ef58-125c-4933-bc7d-c03d3647de23\") "
Sep 16 20:57:48.873020 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.872695    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-home" (OuterVolumeSpecName: "home") pod "2e72ef58-125c-4933-bc7d-c03d3647de23" (UID: "2e72ef58-125c-4933-bc7d-c03d3647de23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:48.873020 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.872728    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e72ef58-125c-4933-bc7d-c03d3647de23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e72ef58-125c-4933-bc7d-c03d3647de23" (UID: "2e72ef58-125c-4933-bc7d-c03d3647de23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:57:48.873898 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.873207    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e72ef58-125c-4933-bc7d-c03d3647de23" (UID: "2e72ef58-125c-4933-bc7d-c03d3647de23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:48.875676 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.875629    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e72ef58-125c-4933-bc7d-c03d3647de23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e72ef58-125c-4933-bc7d-c03d3647de23" (UID: "2e72ef58-125c-4933-bc7d-c03d3647de23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:57:48.876029 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.875998    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-dshm" (OuterVolumeSpecName: "dshm") pod "2e72ef58-125c-4933-bc7d-c03d3647de23" (UID: "2e72ef58-125c-4933-bc7d-c03d3647de23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:48.877803 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.877775    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e72ef58-125c-4933-bc7d-c03d3647de23-kube-api-access-glv4j" (OuterVolumeSpecName: "kube-api-access-glv4j") pod "2e72ef58-125c-4933-bc7d-c03d3647de23" (UID: "2e72ef58-125c-4933-bc7d-c03d3647de23"). InnerVolumeSpecName "kube-api-access-glv4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:57:48.887304 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.887247    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e72ef58-125c-4933-bc7d-c03d3647de23" (UID: "2e72ef58-125c-4933-bc7d-c03d3647de23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:48.903126 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.903089    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e72ef58-125c-4933-bc7d-c03d3647de23" (UID: "2e72ef58-125c-4933-bc7d-c03d3647de23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:57:48.973052 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.973013    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:57:48.973052 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.973048    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:57:48.973052 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.973057    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glv4j\" (UniqueName: \"kubernetes.io/projected/2e72ef58-125c-4933-bc7d-c03d3647de23-kube-api-access-glv4j\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:57:48.973322 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.973067    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e72ef58-125c-4933-bc7d-c03d3647de23-cabundle-cert\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:57:48.973322 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.973076    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e72ef58-125c-4933-bc7d-c03d3647de23-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:57:48.973322 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.973084    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-kserve-provision-location\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:57:48.973322 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.973092    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:57:48.973322 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:48.973102    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e72ef58-125c-4933-bc7d-c03d3647de23-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:57:49.152611 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.152512    2574 generic.go:356] "Generic (PLEG): container finished" podID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerID="a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda" exitCode=0
Sep 16 20:57:49.152611 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.152557    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"
Sep 16 20:57:49.153059 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.152602    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" event={"ID":"2e72ef58-125c-4933-bc7d-c03d3647de23","Type":"ContainerDied","Data":"a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda"}
Sep 16 20:57:49.153059 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.152657    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps" event={"ID":"2e72ef58-125c-4933-bc7d-c03d3647de23","Type":"ContainerDied","Data":"7ae7cdfa630b1c825baa3916ff3fda92a9fdb04c358e9fb26fd2f45f546b5f90"}
Sep 16 20:57:49.153059 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.152681    2574 scope.go:117] "RemoveContainer" containerID="a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda"
Sep 16 20:57:49.161905 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.161882    2574 scope.go:117] "RemoveContainer" containerID="8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91"
Sep 16 20:57:49.170613 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.170578    2574 scope.go:117] "RemoveContainer" containerID="a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda"
Sep 16 20:57:49.170922 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:57:49.170902    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda\": container with ID starting with a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda not found: ID does not exist" containerID="a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda"
Sep 16 20:57:49.171020 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.170935    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda"} err="failed to get container status \"a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda\": rpc error: code = NotFound desc = could not find container \"a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda\": container with ID starting with a8368de5e9d4270c1b85312eee1a4763c73c653e8f32867064e8d8f6898d5eda not found: ID does not exist"
Sep 16 20:57:49.171020 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.170960    2574 scope.go:117] "RemoveContainer" containerID="8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91"
Sep 16 20:57:49.171290 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:57:49.171268    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91\": container with ID starting with 8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91 not found: ID does not exist" containerID="8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91"
Sep 16 20:57:49.171330 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.171296    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91"} err="failed to get container status \"8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91\": rpc error: code = NotFound desc = could not find container \"8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91\": container with ID starting with 8a3013ab47a43fc20c9b4c9ba0b93d811cee686c59c73f7b2e24d91a13680e91 not found: ID does not exist"
Sep 16 20:57:49.171384 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.171367    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"]
Sep 16 20:57:49.174804 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:49.174782    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5c5b674df-qkjps"]
Sep 16 20:57:51.089732 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:57:51.089693    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" path="/var/lib/kubelet/pods/2e72ef58-125c-4933-bc7d-c03d3647de23/volumes"
Sep 16 20:58:14.001036 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:14.000997    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:58:14.054315 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:14.054281    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:58:15.515028 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.514987    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"]
Sep 16 20:58:15.515468 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.515338    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="storage-initializer"
Sep 16 20:58:15.515468 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.515350    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="storage-initializer"
Sep 16 20:58:15.515468 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.515374    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main"
Sep 16 20:58:15.515468 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.515379    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main"
Sep 16 20:58:15.515468 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.515453    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e72ef58-125c-4933-bc7d-c03d3647de23" containerName="main"
Sep 16 20:58:15.518633 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.518614    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.522215 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.522191    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:58:15.522359 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.522285    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:58:15.522434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.522368    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-d8rcr\""
Sep 16 20:58:15.522434 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.522412    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:58:15.528065 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.528039    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"]
Sep 16 20:58:15.602053 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.602002    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bjlt\" (UniqueName: \"kubernetes.io/projected/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-kube-api-access-2bjlt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.602053 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.602055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.602326 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.602124    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.602326 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.602156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.602326 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.602245    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.602326 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.602278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.703045 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.702998    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.703199 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.703055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.703199 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.703082    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bjlt\" (UniqueName: \"kubernetes.io/projected/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-kube-api-access-2bjlt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.703199 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.703100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.703367 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.703199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.703367 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.703241    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.703594 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.703572    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.703661 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.703636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.703716 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.703654    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.705586 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.705542    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.705791 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.705773    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.711902 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.711877    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bjlt\" (UniqueName: \"kubernetes.io/projected/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-kube-api-access-2bjlt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-4r24f\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:15.830606 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:15.830474    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:16.109231 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:16.109202    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"]
Sep 16 20:58:16.110884 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:58:16.110861    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff62a92_4c89_434c_a8c8_9bf2896f5d05.slice/crio-06e791b19404c5c5bb3b459130034fd23d811dc8ca0b1e5d7b29ddec27568bcd WatchSource:0}: Error finding container 06e791b19404c5c5bb3b459130034fd23d811dc8ca0b1e5d7b29ddec27568bcd: Status 404 returned error can't find the container with id 06e791b19404c5c5bb3b459130034fd23d811dc8ca0b1e5d7b29ddec27568bcd
Sep 16 20:58:16.307419 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:16.307348    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f" event={"ID":"0ff62a92-4c89-434c-a8c8-9bf2896f5d05","Type":"ContainerStarted","Data":"a9ed164942ef553bf0573381c968a3790309618f1d8623fa75d88164d35d903a"}
Sep 16 20:58:16.307601 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:16.307466    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f" event={"ID":"0ff62a92-4c89-434c-a8c8-9bf2896f5d05","Type":"ContainerStarted","Data":"06e791b19404c5c5bb3b459130034fd23d811dc8ca0b1e5d7b29ddec27568bcd"}
Sep 16 20:58:16.326479 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:16.326423    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f" podStartSLOduration=1.326396458 podStartE2EDuration="1.326396458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:58:16.324356582 +0000 UTC m=+4727.881701686" watchObservedRunningTime="2026-09-16 20:58:16.326396458 +0000 UTC m=+4727.883741544"
Sep 16 20:58:16.830884 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:16.830840    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:25.154128 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.154093    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"]
Sep 16 20:58:25.157730 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.157701    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:58:25.165635 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:58:25.165590    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:58:25.165635 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:58:25.165635 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:58:25.165635 ip-10-0-141-26 kubenswrapper[2574]:  > containerID="a9ed164942ef553bf0573381c968a3790309618f1d8623fa75d88164d35d903a" cmd=["/bin/sleep","15"]
Sep 16 20:58:25.165971 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.165652    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f" message="" err=<
Sep 16 20:58:25.165971 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:58:25.165971 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:58:25.165971 ip-10-0-141-26 kubenswrapper[2574]:  >
Sep 16 20:58:25.165971 ip-10-0-141-26 kubenswrapper[2574]: E0916 20:58:25.165668    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:58:25.165971 ip-10-0-141-26 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:58:25.165971 ip-10-0-141-26 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Sep 16 20:58:25.165971 ip-10-0-141-26 kubenswrapper[2574]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f" podUID="0ff62a92-4c89-434c-a8c8-9bf2896f5d05" containerName="main" containerID="cri-o://a9ed164942ef553bf0573381c968a3790309618f1d8623fa75d88164d35d903a"
Sep 16 20:58:25.165971 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.165728    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f" podUID="0ff62a92-4c89-434c-a8c8-9bf2896f5d05" containerName="main" containerID="cri-o://a9ed164942ef553bf0573381c968a3790309618f1d8623fa75d88164d35d903a" gracePeriod=60
Sep 16 20:58:25.213637 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.213606    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:58:25.358746 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.358700    2574 generic.go:356] "Generic (PLEG): container finished" podID="0ff62a92-4c89-434c-a8c8-9bf2896f5d05" containerID="a9ed164942ef553bf0573381c968a3790309618f1d8623fa75d88164d35d903a" exitCode=0
Sep 16 20:58:25.358911 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.358770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f" event={"ID":"0ff62a92-4c89-434c-a8c8-9bf2896f5d05","Type":"ContainerDied","Data":"a9ed164942ef553bf0573381c968a3790309618f1d8623fa75d88164d35d903a"}
Sep 16 20:58:25.419153 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.419087    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:25.490780 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.490744    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tls-certs\") pod \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") "
Sep 16 20:58:25.490957 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.490808    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bjlt\" (UniqueName: \"kubernetes.io/projected/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-kube-api-access-2bjlt\") pod \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") "
Sep 16 20:58:25.490957 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.490834    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-home\") pod \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") "
Sep 16 20:58:25.490957 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.490927    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-model-cache\") pod \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") "
Sep 16 20:58:25.491123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.490970    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-dshm\") pod \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") "
Sep 16 20:58:25.491123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.491043    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tmp-dir\") pod \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\" (UID: \"0ff62a92-4c89-434c-a8c8-9bf2896f5d05\") "
Sep 16 20:58:25.491123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.491101    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-home" (OuterVolumeSpecName: "home") pod "0ff62a92-4c89-434c-a8c8-9bf2896f5d05" (UID: "0ff62a92-4c89-434c-a8c8-9bf2896f5d05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:25.491332 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.491227    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ff62a92-4c89-434c-a8c8-9bf2896f5d05" (UID: "0ff62a92-4c89-434c-a8c8-9bf2896f5d05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:25.491393 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.491342    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ff62a92-4c89-434c-a8c8-9bf2896f5d05" (UID: "0ff62a92-4c89-434c-a8c8-9bf2896f5d05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:25.491466 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.491429    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-model-cache\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.491466 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.491448    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-home\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.493209 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.493181    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ff62a92-4c89-434c-a8c8-9bf2896f5d05" (UID: "0ff62a92-4c89-434c-a8c8-9bf2896f5d05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:58:25.493338 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.493208    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-dshm" (OuterVolumeSpecName: "dshm") pod "0ff62a92-4c89-434c-a8c8-9bf2896f5d05" (UID: "0ff62a92-4c89-434c-a8c8-9bf2896f5d05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:58:25.493338 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.493286    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-kube-api-access-2bjlt" (OuterVolumeSpecName: "kube-api-access-2bjlt") pod "0ff62a92-4c89-434c-a8c8-9bf2896f5d05" (UID: "0ff62a92-4c89-434c-a8c8-9bf2896f5d05"). InnerVolumeSpecName "kube-api-access-2bjlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:58:25.592848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.592806    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bjlt\" (UniqueName: \"kubernetes.io/projected/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-kube-api-access-2bjlt\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.592848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.592840    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-dshm\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.592848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.592849    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tmp-dir\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:58:25.592848 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:25.592857    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ff62a92-4c89-434c-a8c8-9bf2896f5d05-tls-certs\") on node \"ip-10-0-141-26.ec2.internal\" DevicePath \"\""
Sep 16 20:58:26.365505 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:26.365466    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f" event={"ID":"0ff62a92-4c89-434c-a8c8-9bf2896f5d05","Type":"ContainerDied","Data":"06e791b19404c5c5bb3b459130034fd23d811dc8ca0b1e5d7b29ddec27568bcd"}
Sep 16 20:58:26.365505 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:26.365501    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"
Sep 16 20:58:26.365994 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:26.365542    2574 scope.go:117] "RemoveContainer" containerID="a9ed164942ef553bf0573381c968a3790309618f1d8623fa75d88164d35d903a"
Sep 16 20:58:26.387281 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:26.387247    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"]
Sep 16 20:58:26.390732 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:26.390704    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-4r24f"]
Sep 16 20:58:27.088860 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:58:27.088826    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ff62a92-4c89-434c-a8c8-9bf2896f5d05" path="/var/lib/kubelet/pods/0ff62a92-4c89-434c-a8c8-9bf2896f5d05/volumes"
Sep 16 20:59:00.765283 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:00.765202    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:06.772385 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:06.772344    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:12.387601 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:12.387567    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:12.584981 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:12.584947    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:12.685786 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:12.685682    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:16.460151 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:16.460115    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:16.515771 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:16.515737    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:18.671623 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:18.671586    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:24.402281 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:24.402241    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:24.503131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:24.503092    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:29.389002 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:29.388972    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:59:29.407702 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:29.407676    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 20:59:39.623481 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:39.623433    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kd9zp_899dc0d4-c2d7-464a-ba4e-25bca3222d2b/discovery/0.log"
Sep 16 20:59:39.637003 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:39.636966    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/0.log"
Sep 16 20:59:39.651874 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:39.651848    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/1.log"
Sep 16 20:59:40.420563 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:40.420527    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kd9zp_899dc0d4-c2d7-464a-ba4e-25bca3222d2b/discovery/0.log"
Sep 16 20:59:40.439331 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:40.439304    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/0.log"
Sep 16 20:59:40.454569 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:40.454544    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/1.log"
Sep 16 20:59:41.818731 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:41.818697    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-lxjs5_c725f6be-302d-4e34-a289-c832d7559508/manager/0.log"
Sep 16 20:59:41.831981 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:41.831942    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-bc6tz_e89b4603-bfbd-41be-8d75-27104dedd7cb/manager/0.log"
Sep 16 20:59:42.019336 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:42.019296    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vnbr5_9514f9df-d0c1-420c-842d-c575c8ddd0a4/manager/0.log"
Sep 16 20:59:42.028372 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:42.028350    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-d9l9h_5acbeed4-2a5e-44ad-8946-8faa44367bf5/limitador/0.log"
Sep 16 20:59:47.941123 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:47.941081    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:48.078922 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.078887    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9wkzj/must-gather-nrbpm"]
Sep 16 20:59:48.079242 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.079230    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff62a92-4c89-434c-a8c8-9bf2896f5d05" containerName="main"
Sep 16 20:59:48.079288 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.079244    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff62a92-4c89-434c-a8c8-9bf2896f5d05" containerName="main"
Sep 16 20:59:48.079327 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.079311    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff62a92-4c89-434c-a8c8-9bf2896f5d05" containerName="main"
Sep 16 20:59:48.082435 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.082396    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wkzj/must-gather-nrbpm"
Sep 16 20:59:48.086342 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.086314    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9wkzj\"/\"kube-root-ca.crt\""
Sep 16 20:59:48.086529 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.086512    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9wkzj\"/\"openshift-service-ca.crt\""
Sep 16 20:59:48.088058 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.088038    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9wkzj\"/\"default-dockercfg-qhzj8\""
Sep 16 20:59:48.102411 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.102376    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wkzj/must-gather-nrbpm"]
Sep 16 20:59:48.106939 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.106915    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:48.206863 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.206777    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/614dd332-525a-424c-bd05-d471463a88bd-must-gather-output\") pod \"must-gather-nrbpm\" (UID: \"614dd332-525a-424c-bd05-d471463a88bd\") " pod="openshift-must-gather-9wkzj/must-gather-nrbpm"
Sep 16 20:59:48.206863 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.206830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djf5h\" (UniqueName: \"kubernetes.io/projected/614dd332-525a-424c-bd05-d471463a88bd-kube-api-access-djf5h\") pod \"must-gather-nrbpm\" (UID: \"614dd332-525a-424c-bd05-d471463a88bd\") " pod="openshift-must-gather-9wkzj/must-gather-nrbpm"
Sep 16 20:59:48.208690 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.208663    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:48.307784 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.307749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djf5h\" (UniqueName: \"kubernetes.io/projected/614dd332-525a-424c-bd05-d471463a88bd-kube-api-access-djf5h\") pod \"must-gather-nrbpm\" (UID: \"614dd332-525a-424c-bd05-d471463a88bd\") " pod="openshift-must-gather-9wkzj/must-gather-nrbpm"
Sep 16 20:59:48.307953 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.307866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/614dd332-525a-424c-bd05-d471463a88bd-must-gather-output\") pod \"must-gather-nrbpm\" (UID: \"614dd332-525a-424c-bd05-d471463a88bd\") " pod="openshift-must-gather-9wkzj/must-gather-nrbpm"
Sep 16 20:59:48.308197 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.308177    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/614dd332-525a-424c-bd05-d471463a88bd-must-gather-output\") pod \"must-gather-nrbpm\" (UID: \"614dd332-525a-424c-bd05-d471463a88bd\") " pod="openshift-must-gather-9wkzj/must-gather-nrbpm"
Sep 16 20:59:48.316733 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.316712    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djf5h\" (UniqueName: \"kubernetes.io/projected/614dd332-525a-424c-bd05-d471463a88bd-kube-api-access-djf5h\") pod \"must-gather-nrbpm\" (UID: \"614dd332-525a-424c-bd05-d471463a88bd\") " pod="openshift-must-gather-9wkzj/must-gather-nrbpm"
Sep 16 20:59:48.402330 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.402297    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wkzj/must-gather-nrbpm"
Sep 16 20:59:48.675244 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.675215    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wkzj/must-gather-nrbpm"]
Sep 16 20:59:48.677052 ip-10-0-141-26 kubenswrapper[2574]: W0916 20:59:48.677024    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod614dd332_525a_424c_bd05_d471463a88bd.slice/crio-8cff35d6ad636ca03b40ec0adae38d597cc2c31e05cff57b97d231731f267dec WatchSource:0}: Error finding container 8cff35d6ad636ca03b40ec0adae38d597cc2c31e05cff57b97d231731f267dec: Status 404 returned error can't find the container with id 8cff35d6ad636ca03b40ec0adae38d597cc2c31e05cff57b97d231731f267dec
Sep 16 20:59:48.679034 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.679011    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:59:48.784252 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:48.784204    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wkzj/must-gather-nrbpm" event={"ID":"614dd332-525a-424c-bd05-d471463a88bd","Type":"ContainerStarted","Data":"8cff35d6ad636ca03b40ec0adae38d597cc2c31e05cff57b97d231731f267dec"}
Sep 16 20:59:49.792143 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:49.792115    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wkzj/must-gather-nrbpm" event={"ID":"614dd332-525a-424c-bd05-d471463a88bd","Type":"ContainerStarted","Data":"f295e451137f586c7b2666e8eb72d1e4c80f648f1813f0d46b3f925490d59308"}
Sep 16 20:59:49.792526 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:49.792154    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wkzj/must-gather-nrbpm" event={"ID":"614dd332-525a-424c-bd05-d471463a88bd","Type":"ContainerStarted","Data":"e4a26a043fff9a6dd0ba434b267c039f60436adc1d57bd659f6a9105d5c90674"}
Sep 16 20:59:49.809368 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:49.809305    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9wkzj/must-gather-nrbpm" podStartSLOduration=0.913402501 podStartE2EDuration="1.809282939s" totalImagesPullingTime="895.880438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:59:49.808842066 +0000 UTC m=+4821.366187167" watchObservedRunningTime="2026-09-16 20:59:49.809282939 +0000 UTC m=+4821.366628027"
Sep 16 20:59:51.113074 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:51.113040    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jp5lb_c883cd9b-80f1-47f4-a8dc-6ae0c1eacdaa/global-pull-secret-syncer/0.log"
Sep 16 20:59:51.224414 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:51.224366    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log"
Sep 16 20:59:51.394477 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:51.394376    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-26.ec2.internal_d3f4231defbbc87c04f877cedab58518/haproxy/0.log"
Sep 16 20:59:53.878131 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:53.878085    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:53.931388 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:53.931355    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lcb2q_c11e1d7a-2544-4453-9d6c-6f306de66d66/konnectivity-agent/0.log" limit=1024
Sep 16 20:59:56.760161 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:56.760103    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-lxjs5_c725f6be-302d-4e34-a289-c832d7559508/manager/0.log"
Sep 16 20:59:56.801117 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:56.801088    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-bc6tz_e89b4603-bfbd-41be-8d75-27104dedd7cb/manager/0.log"
Sep 16 20:59:57.080742 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:57.080634    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-vnbr5_9514f9df-d0c1-420c-842d-c575c8ddd0a4/manager/0.log"
Sep 16 20:59:57.123457 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:57.123424    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-d9l9h_5acbeed4-2a5e-44ad-8946-8faa44367bf5/limitador/0.log"
Sep 16 20:59:58.471009 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:58.470913    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2jj6n_7b4b6cbf-5346-4264-874a-e299be9066b0/kube-state-metrics/0.log"
Sep 16 20:59:58.503760 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:58.502637    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2jj6n_7b4b6cbf-5346-4264-874a-e299be9066b0/kube-rbac-proxy-main/0.log"
Sep 16 20:59:58.530842 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:58.530805    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-2jj6n_7b4b6cbf-5346-4264-874a-e299be9066b0/kube-rbac-proxy-self/0.log"
Sep 16 20:59:58.570074 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:58.569501    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b8f869fd6-4vlm4_4a16effe-fc2f-4cf8-a2bd-aebb5a991442/metrics-server/0.log"
Sep 16 20:59:58.659070 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:58.659039    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcgb9_53b9e6b6-25f6-4718-84bd-93941f725f7c/node-exporter/0.log"
Sep 16 20:59:58.684235 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:58.684204    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcgb9_53b9e6b6-25f6-4718-84bd-93941f725f7c/kube-rbac-proxy/0.log"
Sep 16 20:59:58.706951 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:58.706920    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcgb9_53b9e6b6-25f6-4718-84bd-93941f725f7c/init-textfile/0.log"
Sep 16 20:59:59.151857 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:59.151823    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-9d9rg_cf912f13-579e-4f8f-86b1-1d7795227c93/prometheus-operator/0.log"
Sep 16 20:59:59.170298 ip-10-0-141-26 kubenswrapper[2574]: I0916 20:59:59.170190    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-9d9rg_cf912f13-579e-4f8f-86b1-1d7795227c93/kube-rbac-proxy/0.log"
Sep 16 21:00:01.028476 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.028443    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/2.log"
Sep 16 21:00:01.034730 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.034701    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-rq9x9_eb3b9b20-998e-47b3-a336-46904530aa4e/console-operator/3.log"
Sep 16 21:00:01.181862 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.181828    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"]
Sep 16 21:00:01.186101 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.186071    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.195810 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.195757    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"]
Sep 16 21:00:01.335797 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.335761    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-sys\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.336086 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.336045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-podres\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.336292 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.336273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dsd9\" (UniqueName: \"kubernetes.io/projected/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-kube-api-access-5dsd9\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.336503 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.336483    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-lib-modules\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.336716 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.336698    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-proc\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.438103 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.438069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dsd9\" (UniqueName: \"kubernetes.io/projected/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-kube-api-access-5dsd9\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.438305 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.438123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-lib-modules\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.438305 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.438183    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-proc\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.438305 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.438219    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-sys\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.438305 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.438243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-podres\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.438569 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.438358    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-lib-modules\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.438569 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.438387    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-podres\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.438569 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.438454    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-proc\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.438569 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.438464    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-sys\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.446626 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.446597    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dsd9\" (UniqueName: \"kubernetes.io/projected/be913e8d-08ab-402e-a8c7-893ffd2b3fdc-kube-api-access-5dsd9\") pod \"perf-node-gather-daemonset-q89tm\" (UID: \"be913e8d-08ab-402e-a8c7-893ffd2b3fdc\") " pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.499459 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.499427    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.501978 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.501950    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84fc8d9d69-mx87v_e87100aa-cd82-4865-9f68-2bf2bd1089fd/console/0.log"
Sep 16 21:00:01.777628 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.777594    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"]
Sep 16 21:00:01.780631 ip-10-0-141-26 kubenswrapper[2574]: W0916 21:00:01.780601    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbe913e8d_08ab_402e_a8c7_893ffd2b3fdc.slice/crio-c8ba9b24d81664d50a903172e31c744cd9e7f359a81beee115200b5407c219fa WatchSource:0}: Error finding container c8ba9b24d81664d50a903172e31c744cd9e7f359a81beee115200b5407c219fa: Status 404 returned error can't find the container with id c8ba9b24d81664d50a903172e31c744cd9e7f359a81beee115200b5407c219fa
Sep 16 21:00:01.880971 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.880924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm" event={"ID":"be913e8d-08ab-402e-a8c7-893ffd2b3fdc","Type":"ContainerStarted","Data":"39298d33d8a3900eac19c0485aa0cc7d75311792898b28e83886b87dda7d497a"}
Sep 16 21:00:01.881172 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.880990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm" event={"ID":"be913e8d-08ab-402e-a8c7-893ffd2b3fdc","Type":"ContainerStarted","Data":"c8ba9b24d81664d50a903172e31c744cd9e7f359a81beee115200b5407c219fa"}
Sep 16 21:00:01.881172 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.881057    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:01.898770 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:01.898709    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm" podStartSLOduration=0.898690804 podStartE2EDuration="898.690804ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:00:01.896656514 +0000 UTC m=+4833.454001624" watchObservedRunningTime="2026-09-16 21:00:01.898690804 +0000 UTC m=+4833.456035943"
Sep 16 21:00:02.122068 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:02.122030    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-5gkls_7b26210c-13af-4735-8217-764768cd4927/volume-data-source-validator/0.log"
Sep 16 21:00:03.106801 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:03.106766    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5qm2f_01b5832d-8fc0-467b-b874-1a22e958fb24/dns/0.log"
Sep 16 21:00:03.127304 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:03.127257    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5qm2f_01b5832d-8fc0-467b-b874-1a22e958fb24/kube-rbac-proxy/0.log"
Sep 16 21:00:03.292598 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:03.292573    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n6mm2_d577c619-6162-4417-8e78-f3bff26da8b7/dns-node-resolver/0.log"
Sep 16 21:00:03.827918 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:03.827885    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d5d7d58c7-6q5zw_1287efa8-a543-4625-b434-b92369c9c632/registry/0.log"
Sep 16 21:00:03.896673 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:03.896645    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jdb2d_8f185472-cb62-407e-8507-df45a42c37e2/node-ca/0.log"
Sep 16 21:00:05.017385 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:05.017350    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kd9zp_899dc0d4-c2d7-464a-ba4e-25bca3222d2b/discovery/0.log"
Sep 16 21:00:05.045298 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:05.045265    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/0.log"
Sep 16 21:00:05.065895 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:05.065857    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/1.log"
Sep 16 21:00:05.626435 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:05.626382    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vzhbl_025c82f2-833e-4969-8602-ef8b0c6c8b64/serve-healthcheck-canary/0.log"
Sep 16 21:00:06.154907 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:06.154880    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-42phd_cbe68db7-0031-4a44-b63c-4b30698c5663/kube-rbac-proxy/0.log"
Sep 16 21:00:06.177433 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:06.177390    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-42phd_cbe68db7-0031-4a44-b63c-4b30698c5663/exporter/0.log"
Sep 16 21:00:06.200979 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:06.200945    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-42phd_cbe68db7-0031-4a44-b63c-4b30698c5663/extractor/0.log"
Sep 16 21:00:07.900576 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:07.900548    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9wkzj/perf-node-gather-daemonset-q89tm"
Sep 16 21:00:10.830179 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:10.830151    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kd9zp_899dc0d4-c2d7-464a-ba4e-25bca3222d2b/discovery/0.log"
Sep 16 21:00:10.847481 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:10.847448    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/0.log"
Sep 16 21:00:10.871982 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:10.871937    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/1.log"
Sep 16 21:00:12.070384 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:12.070353    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-kd9zp_899dc0d4-c2d7-464a-ba4e-25bca3222d2b/discovery/0.log"
Sep 16 21:00:12.088049 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:12.088018    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/0.log"
Sep 16 21:00:12.111532 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:12.111506    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-z4fth_2097f2b3-4c75-432b-9854-5d745a938aa1/istio-proxy/1.log"
Sep 16 21:00:17.352729 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:17.352676    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-ggc42_cd56b036-b438-45fa-9e25-26ab878f80da/kube-storage-version-migrator-operator/1.log"
Sep 16 21:00:17.353496 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:17.353476    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-ggc42_cd56b036-b438-45fa-9e25-26ab878f80da/kube-storage-version-migrator-operator/0.log"
Sep 16 21:00:18.549536 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.549507    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxm92_ecc37a3c-9ad5-48c3-858e-31e943c27f92/kube-multus-additional-cni-plugins/0.log"
Sep 16 21:00:18.571990 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.571960    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxm92_ecc37a3c-9ad5-48c3-858e-31e943c27f92/egress-router-binary-copy/0.log"
Sep 16 21:00:18.594043 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.593992    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxm92_ecc37a3c-9ad5-48c3-858e-31e943c27f92/cni-plugins/0.log"
Sep 16 21:00:18.614840 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.614814    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxm92_ecc37a3c-9ad5-48c3-858e-31e943c27f92/bond-cni-plugin/0.log"
Sep 16 21:00:18.635272 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.635251    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxm92_ecc37a3c-9ad5-48c3-858e-31e943c27f92/routeoverride-cni/0.log"
Sep 16 21:00:18.658914 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.658892    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxm92_ecc37a3c-9ad5-48c3-858e-31e943c27f92/whereabouts-cni-bincopy/0.log"
Sep 16 21:00:18.679780 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.679752    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxm92_ecc37a3c-9ad5-48c3-858e-31e943c27f92/whereabouts-cni/0.log"
Sep 16 21:00:18.875275 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.875245    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bwh92_82fad23c-10de-49aa-b40d-0d3b45360cbb/kube-multus/0.log"
Sep 16 21:00:18.901151 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.901114    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9fgnw_4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825/network-metrics-daemon/0.log"
Sep 16 21:00:18.916137 ip-10-0-141-26 kubenswrapper[2574]: I0916 21:00:18.916110    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9fgnw_4ee25f30-64ca-4a7c-a3e4-d98bfbe3c825/kube-rbac-proxy/0.log"