Jul 28 17:08:27.414192 ip-10-0-137-238 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:08:27.414203 ip-10-0-137-238 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:08:27.414209 ip-10-0-137-238 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:08:27.414424 ip-10-0-137-238 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:08:38.761120 ip-10-0-137-238 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:08:38.761135 ip-10-0-137-238 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b3780618247343f4a2691ecbe061e979 --
Jul 28 17:10:33.664694 ip-10-0-137-238 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:10:33.995784 ip-10-0-137-238 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:33.995784 ip-10-0-137-238 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:10:33.995784 ip-10-0-137-238 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:33.995784 ip-10-0-137-238 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:10:33.995784 ip-10-0-137-238 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:33.997495 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:33.997419    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:10:34.001571 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001554    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:34.001571 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001569    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:34.001571 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001574    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001578    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001583    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001587    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001591    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001595    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001598    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001603    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001607    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001611    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001615    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001619    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001624    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001628    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001632    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001635    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001640    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001645    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001649    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001653    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:34.001749 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001657    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001661    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001665    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001669    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001673    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001677    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001681    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001686    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001690    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001694    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001699    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001704    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001710    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001718    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001723    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001728    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001732    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001736    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001741    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001747    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:34.002388 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001752    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001756    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001761    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001765    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001769    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001773    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001777    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001781    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001785    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001789    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001794    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001798    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001802    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001806    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001810    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001814    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001821    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001827    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001832    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:34.003071 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001836    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001841    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001845    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001849    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001853    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001858    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001864    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001869    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001873    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001877    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001883    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001887    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001892    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001898    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001903    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001908    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001912    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001917    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001921    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001925    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:34.003940 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001929    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001934    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001938    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001942    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001946    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001950    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001955    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001959    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001963    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001967    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001971    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001976    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001980    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001984    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001988    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001992    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.001997    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.002001    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003743    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:34.004799 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003755    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003760    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003765    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003770    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003775    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003780    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003784    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003789    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003793    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003797    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003801    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003806    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003812    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003818    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003823    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003828    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003832    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003836    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003841    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003845    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:34.005625 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003849    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003853    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003857    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003861    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003872    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003877    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003881    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003885    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003890    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003894    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003898    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003902    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003907    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003912    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003916    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003920    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003924    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003929    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003933    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003937    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:34.006519 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003941    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003945    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003949    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003952    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003957    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003961    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003965    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003969    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003973    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003977    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003982    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003987    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003991    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003995    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.003999    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004004    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004008    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004012    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004016    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004044    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:34.007135 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004050    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004054    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004059    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004064    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004068    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004073    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004077    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004082    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004090    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004096    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004101    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004106    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004110    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004114    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004119    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004124    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004130    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004135    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004140    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:34.007888 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004144    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004148    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004153    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004157    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004161    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004165    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004169    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004173    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004178    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004182    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004186    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004190    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004195    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004199    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004203    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004208    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004211    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004215    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004221    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:34.008533 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004810    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004819    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004825    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004831    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004835    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004841    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004845    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004849    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004853    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004858    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004862    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004866    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004870    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004874    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004878    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004883    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004887    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004891    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004895    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004899    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:34.009011 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004903    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004908    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004912    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004916    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004920    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004925    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004929    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004933    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004937    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004941    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004945    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004950    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004954    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004960    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004965    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004969    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004973    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004978    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004982    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:34.009696 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004986    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004990    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004994    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.004998    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005002    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005008    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005012    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005017    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005021    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005025    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005029    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005033    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005037    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005045    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005050    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005054    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005058    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005062    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005066    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005071    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:34.010268 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005075    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005079    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005083    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005087    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005091    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005095    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005104    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005108    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005112    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005116    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005120    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005126    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005133    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005138    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005145    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005151    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005157    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005163    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005167    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:34.010806 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005172    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005177    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005181    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005186    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005190    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005194    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005200    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005204    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005208    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005213    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005217    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005221    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005225    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005229    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005233    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005237    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005243    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005247    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005251    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005255    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:34.011379 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.005260    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005354    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005364    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005373    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005380    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005386    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005391    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005415    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005422    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005427    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005432    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005438    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005443    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005448    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005453    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005457    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005462    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005466    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005473    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005482    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005487    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005492    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005497    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005502    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:10:34.011920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005508    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005512    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005517    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005523    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005530    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005534    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005540    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005545    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005549    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005557    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005562    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005566    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005571    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005576    2579 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005581    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005587    2579 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005591    2579 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005596    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005601    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005606    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005612    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005616    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005621    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005625    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005630    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:10:34.012522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005634    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005640    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005645    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005649    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005654    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005659    2579 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005665    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005670    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005674    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005680    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005684    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005694    2579 flags.go:64] FLAG: --help="false"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005698    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005704    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005708    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005713    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005719    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005725    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005730    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005734    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005739    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005744    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005749    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005754    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:10:34.013120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005758    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005763    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005768    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005773    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005777    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005783    2579 flags.go:64] FLAG: --lock-file=""
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005787    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005792    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005797    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005807    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005812    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005816    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005821    2579 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005826    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005831    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005836    2579 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005841    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005847    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005852    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005860    2579 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005865    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005870    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005874    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005879    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005884    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:10:34.013791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005889    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005894    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005905    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005910    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005915    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005919    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005924    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005932    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005937    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005942    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005947    2579 flags.go:64] FLAG: --port="10250"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005951    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005956    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0024f88eff843a9ea"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005961    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005966    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005971    2579 flags.go:64] FLAG: --register-node="true"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005976    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005983    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005988    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005993    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.005998    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006003    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006008    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006013    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006018    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:10:34.014519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006023    2579 flags.go:64] FLAG: --runonce="false"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006029    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006034    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006039    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006043    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006048    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006053    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006057    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006063    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006068    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006073    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006078    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006082    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006087    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006092    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006097    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006105    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006110    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006115    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006121    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006126    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006131    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006135    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006140    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006146    2579 flags.go:64] FLAG: --v="2"
Jul 28 17:10:34.015130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006152    2579 flags.go:64] FLAG: --version="false"
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006158    2579 flags.go:64] FLAG: --vmodule=""
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006164    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.006169    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006337    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006344    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006349    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006353    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006359    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006363    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006367    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006371    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006375    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006379    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006383    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006387    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006392    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006416    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006421    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006425    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006429    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:34.015763 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006434    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006438    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006442    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006447    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006451    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006455    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006459    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006464    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006468    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006472    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006478    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006482    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006486    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006491    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006495    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006499    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006503    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006507    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006511    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:34.016287 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006517    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006521    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006525    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006528    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006532    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006536    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006540    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006544    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006548    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006552    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006557    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006561    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006565    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006569    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006573    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006578    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006582    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006586    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006591    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006595    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:34.016759 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006599    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006603    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006608    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006612    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006616    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006620    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006624    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006628    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006632    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006636    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006640    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006644    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006653    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006658    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006663    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006668    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006671    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006675    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006679    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:34.017234 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006684    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006689    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006695    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006701    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006706    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006710    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006714    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006718    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006725    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006730    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006734    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006738    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006742    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006746    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006751    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006755    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006759    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006763    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006767    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:34.017714 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006771    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006775    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006779    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006783    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.006787    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.007455    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.015345    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.015359    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.015993    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.016001    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016053    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016059    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016062    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016065    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016068    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:34.018188 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016071    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016075    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016079    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016083    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016086    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016089    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016091    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016094    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016097    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016099    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016102    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016104    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016107    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016110    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016113    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016115    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016118    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016121    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016123    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016126    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:34.018620 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016129    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016131    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016134    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016137    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016141    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016144    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016146    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016149    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016152    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016155    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016157    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016160    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016162    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016165    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016168    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016171    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016174    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016176    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016179    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016181    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:34.019101 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016184    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016186    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016189    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016192    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016194    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016197    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016200    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016202    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016205    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016208    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016210    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016213    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016215    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016218    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016222    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016225    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016229    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016232    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016234    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:34.019604 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016237    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016240    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016243    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016246    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016248    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016253    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016256    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016259    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016261    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016264    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016267    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016270    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016273    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016275    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016278    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016281    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016284    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016286    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016289    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:34.020103 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016292    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016295    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016297    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016301    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016303    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016306    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016309    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016311    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016314    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016317    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016319    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016322    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016325    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016327    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016329    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:34.020582 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016332    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.016337    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016463    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016469    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016471    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016474    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016477    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016479    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016482    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016485    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016487    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016490    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016492    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016495    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016498    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:34.020953 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016501    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016503    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016506    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016509    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016511    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016514    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016516    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016519    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016521    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016524    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016528    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016532    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016535    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016538    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016540    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016543    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016545    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016548    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016550    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:34.021323 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016553    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016556    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016558    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016561    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016563    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016566    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016568    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016570    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016573    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016576    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016578    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016581    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016583    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016587    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016591    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016594    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016597    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016599    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016602    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:34.021907 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016605    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016607    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016610    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016612    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016615    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016618    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016620    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016623    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016626    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016628    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016630    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016633    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016635    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016638    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016640    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016643    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016645    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016648    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016650    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:34.022365 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016653    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016655    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016658    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016660    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016663    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016665    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016667    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016670    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016673    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016676    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016678    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016680    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016683    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016685    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016688    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016690    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016693    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016695    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016699    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016703    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:34.022852 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016706    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016709    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016711    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016714    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016716    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016719    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016721    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016724    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:34.016727    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.016731    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.016834    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.018812    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.019986    2579 server.go:1019] "Starting client certificate rotation"
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.020087    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:34.023326 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.020805    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:34.023729 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.022145    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:34.023729 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.023264    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:34.034781 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.034765    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:10:34.037469 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.037453    2579 log.go:25] "Validated CRI v1 image API"
Jul 28 17:10:34.038904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.038889    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:10:34.041081 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.041061    2579 fs.go:135] Filesystem UUIDs: map[63960f7a-409c-429a-b44e-dc8e384f6764:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ebf770ca-eea8-41d4-8574-150a75ea8dff:/dev/nvme0n1p3]
Jul 28 17:10:34.041152 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.041079    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:10:34.045748 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.045638    2579 manager.go:216] Machine: {Timestamp:2026-07-28 17:10:34.044606781 +0000 UTC m=+0.279572224 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095302 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a3e76b0560f78aa708efc16869e17 SystemUUID:ec2a3e76-b056-0f78-aa70-8efc16869e17 BootID:b3780618-2473-43f4-a269-1ecbe061e979 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d2:b9:5f:62:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d2:b9:5f:62:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:b0:90:8e:43:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:10:34.045748 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.045740    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:10:34.045904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.045867    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:10:34.046734 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.046713    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:10:34.046880 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.046736    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-238.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:10:34.047575 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.047563    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:10:34.047628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.047581    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:10:34.047628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.047591    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:34.048215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.048203    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:34.049097 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.049085    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:10:34.051954 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.051935    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:34.052026 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.051982    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:34.052117 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.052106    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:10:34.052177 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.052143    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:10:34.052177 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.052158    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:10:34.052177 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.052172    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:10:34.052294 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.052185    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:10:34.052294 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.052196    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:10:34.052565 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.052551    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:34.052639 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.052574    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:34.054411 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.054386    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:10:34.056192 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.056179    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:10:34.056229 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.056209    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:10:34.057792 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057779    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:10:34.057841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057795    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:10:34.057841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057801    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:10:34.057841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057813    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:10:34.057841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057822    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:10:34.057841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057827    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:10:34.057841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057835    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:10:34.057841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057840    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:10:34.058028 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057847    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:10:34.058028 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057854    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:10:34.058028 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057871    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:10:34.058028 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.057876    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:10:34.058460 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.058450    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:10:34.058460 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.058459    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:10:34.061212 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.061199    2579 server.go:1268] "Started kubelet"
Jul 28 17:10:34.061322 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.061271    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:10:34.061369 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.061346    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:10:34.062025 ip-10-0-137-238 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:10:34.062266 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.062019    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:10:34.063258 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.063235    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:10:34.063941 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.063927    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:10:34.064860 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.064836    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-238.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:10:34.064965 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.064907    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:10:34.066645 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.066592    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:10:34.067004 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.066982    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:10:34.071295 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.071274    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:10:34.071380 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.071296    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:34.071614 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.071593    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.072204 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.071134    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-238.ec2.internal.18c68344db6a5a24  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-238.ec2.internal,UID:ip-10-0-137-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-238.ec2.internal,},FirstTimestamp:2026-07-28 17:10:34.061175332 +0000 UTC m=+0.296140775,LastTimestamp:2026-07-28 17:10:34.061175332 +0000 UTC m=+0.296140775,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-238.ec2.internal,}"
Jul 28 17:10:34.072315 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072239    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:10:34.072315 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072255    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:10:34.072433 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072388    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:10:34.072433 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072413    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:10:34.072521 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072456    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:10:34.072599 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072574    2579 factory.go:55] Registering systemd factory
Jul 28 17:10:34.072599 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072590    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:10:34.072807 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072795    2579 factory.go:153] Registering CRI-O factory
Jul 28 17:10:34.072862 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072811    2579 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:10:34.072862 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072855    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:10:34.072957 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072877    2579 factory.go:103] Registering Raw factory
Jul 28 17:10:34.072957 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.072892    2579 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:10:34.073364 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.073349    2579 manager.go:349] Starting recovery of all containers
Jul 28 17:10:34.076959 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.076926    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-238.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:10:34.077108 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.077087    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:10:34.083952 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.083932    2579 manager.go:354] Recovery completed
Jul 28 17:10:34.087835 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.087823    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:34.089765 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.089743    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:34.089844 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.089774    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:34.089844 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.089789    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:34.090262 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.090251    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:10:34.090262 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.090262    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:10:34.090339 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.090273    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:34.090339 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.090279    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:34.090920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.090905    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:34.090989 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.090931    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:34.090989 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.090944    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:34.092149 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.092081    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-238.ec2.internal.18c68344dd1e7e30  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-238.ec2.internal,UID:ip-10-0-137-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-238.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-238.ec2.internal,},FirstTimestamp:2026-07-28 17:10:34.089758256 +0000 UTC m=+0.324723706,LastTimestamp:2026-07-28 17:10:34.089758256 +0000 UTC m=+0.324723706,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-238.ec2.internal,}"
Jul 28 17:10:34.092977 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.092964    2579 policy_none.go:49] "None policy: Start"
Jul 28 17:10:34.093034 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.092981    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:10:34.093034 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.092994    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:10:34.094932 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.094921    2579 policy_none.go:47] "Start"
Jul 28 17:10:34.103527 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.103460    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-238.ec2.internal.18c68344dd1ed78b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-238.ec2.internal,UID:ip-10-0-137-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-238.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-238.ec2.internal,},FirstTimestamp:2026-07-28 17:10:34.089781131 +0000 UTC m=+0.324746576,LastTimestamp:2026-07-28 17:10:34.089781131 +0000 UTC m=+0.324746576,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-238.ec2.internal,}"
Jul 28 17:10:34.110550 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.110493    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-238.ec2.internal.18c68344dd1f0d21  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-238.ec2.internal,UID:ip-10-0-137-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-238.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-238.ec2.internal,},FirstTimestamp:2026-07-28 17:10:34.089794849 +0000 UTC m=+0.324760293,LastTimestamp:2026-07-28 17:10:34.089794849 +0000 UTC m=+0.324760293,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-238.ec2.internal,}"
Jul 28 17:10:34.115329 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.115274    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-238.ec2.internal.18c68344dd1e7e30\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-238.ec2.internal.18c68344dd1e7e30  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-238.ec2.internal,UID:ip-10-0-137-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-238.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-238.ec2.internal,},FirstTimestamp:2026-07-28 17:10:34.089758256 +0000 UTC m=+0.324723706,LastTimestamp:2026-07-28 17:10:34.090917962 +0000 UTC m=+0.325883405,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-238.ec2.internal,}"
Jul 28 17:10:34.123636 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.123498    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-238.ec2.internal.18c68344dd1ed78b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-238.ec2.internal.18c68344dd1ed78b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-238.ec2.internal,UID:ip-10-0-137-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-238.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-238.ec2.internal,},FirstTimestamp:2026-07-28 17:10:34.089781131 +0000 UTC m=+0.324746576,LastTimestamp:2026-07-28 17:10:34.090937039 +0000 UTC m=+0.325902483,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-238.ec2.internal,}"
Jul 28 17:10:34.131075 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.131060    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hsfxb"
Jul 28 17:10:34.135926 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.135915    2579 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:10:34.135982 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.135941    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:10:34.135982 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.135949    2579 server.go:87] "Starting device plugin registration server"
Jul 28 17:10:34.136126 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.136115    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:10:34.136169 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.136128    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:10:34.136236 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.136208    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:10:34.136318 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.136306    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:10:34.136318 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.136318    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:10:34.137702 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.137424    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:10:34.137702 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.137473    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.137909 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.137827    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-238.ec2.internal.18c68344dd1f0d21\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-238.ec2.internal.18c68344dd1f0d21  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-238.ec2.internal,UID:ip-10-0-137-238.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-238.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-238.ec2.internal,},FirstTimestamp:2026-07-28 17:10:34.089794849 +0000 UTC m=+0.324760293,LastTimestamp:2026-07-28 17:10:34.09094896 +0000 UTC m=+0.325914404,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-238.ec2.internal,}"
Jul 28 17:10:34.139243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.139226    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hsfxb"
Jul 28 17:10:34.236621 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.236599    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:34.238717 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.237355    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:34.238717 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.237381    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:34.238717 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.237394    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:34.238717 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.237441    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.245853 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.245794    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.245853 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.245816    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-238.ec2.internal\": node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.268651 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.268629    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.276919 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.276890    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:10:34.278226 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.278203    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:10:34.278302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.278245    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:10:34.278302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.278267    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:10:34.278385 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.278302    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:10:34.280591 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.280574    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:34.368797 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.368764    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.378928 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.378902    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal"]
Jul 28 17:10:34.378980 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.378970    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:34.380457 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.380434    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:34.380517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.380472    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:34.380517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.380487    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:34.382979 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.382965    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:34.383114 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.383099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.383153 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.383130    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:34.383712 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.383694    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:34.383815 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.383721    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:34.383815 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.383737    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:34.383815 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.383755    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:34.383815 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.383779    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:34.383815 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.383790    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:34.386231 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.386213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.386322 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.386246    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:34.387002 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.386986    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:34.387085 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.387017    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:34.387085 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.387038    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:34.416091 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.416067    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-238.ec2.internal\" not found" node="ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.420751 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.420735    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-238.ec2.internal\" not found" node="ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.468934 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.468907    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.569727 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.569671    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.574384 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.574367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d68380c15b9e00b00440ef022ddd164-config\") pod \"kube-apiserver-proxy-ip-10-0-137-238.ec2.internal\" (UID: \"6d68380c15b9e00b00440ef022ddd164\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.574467 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.574424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.574538 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.574491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.676561 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.676535    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.676644 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.676614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d68380c15b9e00b00440ef022ddd164-config\") pod \"kube-apiserver-proxy-ip-10-0-137-238.ec2.internal\" (UID: \"6d68380c15b9e00b00440ef022ddd164\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.676707 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.676646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.676707 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.676675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.676805 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.676718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d68380c15b9e00b00440ef022ddd164-config\") pod \"kube-apiserver-proxy-ip-10-0-137-238.ec2.internal\" (UID: \"6d68380c15b9e00b00440ef022ddd164\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.676805 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.676727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.676805 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.676723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67a91ffb87b056369246a9445ebaa0cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal\" (UID: \"67a91ffb87b056369246a9445ebaa0cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.722189 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.722168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.724494 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:34.724479    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:34.776756 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.776731    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.878097 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.878039    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:34.978967 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:34.978933    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:35.021305 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.021280    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:10:35.021719 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.021440    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:35.079651 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:35.079626    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:35.145242 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.145163    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:05:34 +0000 UTC" deadline="2028-03-19 00:05:32.146925328 +0000 UTC"
Jul 28 17:10:35.145242 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.145203    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14382h54m57.001732277s"
Jul 28 17:10:35.180243 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:35.180216    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:35.240594 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.240576    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:35.242247 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.242227    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:35.242338 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.242262    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:35.242338 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.242279    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:35.281415 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:35.281377    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:35.319077 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.319051    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:35.371149 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:35.371105    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d68380c15b9e00b00440ef022ddd164.slice/crio-18947e7a7a0b1ed0e5f55cc41cb4bbdef9d71103655c0ecaed13f4957965d27f WatchSource:0}: Error finding container 18947e7a7a0b1ed0e5f55cc41cb4bbdef9d71103655c0ecaed13f4957965d27f: Status 404 returned error can't find the container with id 18947e7a7a0b1ed0e5f55cc41cb4bbdef9d71103655c0ecaed13f4957965d27f
Jul 28 17:10:35.371972 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:35.371942    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a91ffb87b056369246a9445ebaa0cf.slice/crio-8c78668400e34558c9ced2e55575f392b9326ed7b2393bf6bd1bb7f9eb20cdc9 WatchSource:0}: Error finding container 8c78668400e34558c9ced2e55575f392b9326ed7b2393bf6bd1bb7f9eb20cdc9: Status 404 returned error can't find the container with id 8c78668400e34558c9ced2e55575f392b9326ed7b2393bf6bd1bb7f9eb20cdc9
Jul 28 17:10:35.376456 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.376431    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:10:35.381773 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:35.381750    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:35.482293 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:35.482232    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:35.582480 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:35.582459    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-238.ec2.internal\" not found"
Jul 28 17:10:35.588403 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.588380    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:35.666098 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.666076    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:35.672277 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.672260    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:35.685803 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.685782    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:35.686707 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.686694    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal"
Jul 28 17:10:35.696203 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:35.696186    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:36.054366 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.054338    2579 apiserver.go:52] "Watching apiserver"
Jul 28 17:10:36.060933 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.060911    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:10:36.061428 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.061399    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qf77t","kube-system/konnectivity-agent-bfd75","kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf","openshift-cluster-node-tuning-operator/tuned-f5tlf","openshift-dns/node-resolver-7tk7q","openshift-image-registry/node-ca-qkksk","openshift-ovn-kubernetes/ovnkube-node-scdr5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal","openshift-multus/multus-additional-cni-plugins-9sl5f","openshift-multus/multus-rrq8b","openshift-multus/network-metrics-daemon-zvdqk","openshift-network-diagnostics/network-check-target-t542q","openshift-network-operator/iptables-alerter-bw7xm"]
Jul 28 17:10:36.064879 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.064176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.066548 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.066524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:36.066643 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.066616    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:36.068011 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.067987    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:10:36.068251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.068225    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:10:36.068354 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.068270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:10:36.068458 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.068434    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hzjbl\""
Jul 28 17:10:36.069703 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.069162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.072475 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.072449    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:36.072574 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.072538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.072745 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.072724    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:10:36.072986 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.072969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-982pd\""
Jul 28 17:10:36.073180 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.073162    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:36.073248 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.073185    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:36.074199 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.073839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.076412 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.076170    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:10:36.076412 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.076246    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:10:36.076601 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.076442    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:10:36.076601 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.076496    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9q7vj\""
Jul 28 17:10:36.076716 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.076704    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:36.076771 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.076738    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-85wnf\""
Jul 28 17:10:36.077049 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.077020    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:36.077574 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.077555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.079870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.079847    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:10:36.080402 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.080204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ng6cl\""
Jul 28 17:10:36.080402 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.080252    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:10:36.082402 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.081346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.084461 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:36.084563 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/195c1605-870d-4b90-b45d-64b90352396d-iptables-alerter-script\") pod \"iptables-alerter-bw7xm\" (UID: \"195c1605-870d-4b90-b45d-64b90352396d\") " pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.084563 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eba498c7-6988-4f37-bc5b-3668877fde57-etc-tuned\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.084563 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dd162ac-a8b4-4ed7-9537-e37504a11570-host\") pod \"node-ca-qkksk\" (UID: \"8dd162ac-a8b4-4ed7-9537-e37504a11570\") " pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.084742 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084574    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-registration-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.084742 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-etc-selinux\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.084742 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mdh\" (UniqueName: \"kubernetes.io/projected/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-kube-api-access-z5mdh\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.084742 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8dd162ac-a8b4-4ed7-9537-e37504a11570-serviceca\") pod \"node-ca-qkksk\" (UID: \"8dd162ac-a8b4-4ed7-9537-e37504a11570\") " pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.084742 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7v4r\" (UniqueName: \"kubernetes.io/projected/195c1605-870d-4b90-b45d-64b90352396d-kube-api-access-q7v4r\") pod \"iptables-alerter-bw7xm\" (UID: \"195c1605-870d-4b90-b45d-64b90352396d\") " pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-sys-fs\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-sysconfig\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-sysctl-conf\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-lib-modules\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkcsx\" (UniqueName: \"kubernetes.io/projected/eba498c7-6988-4f37-bc5b-3668877fde57-kube-api-access-xkcsx\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195c1605-870d-4b90-b45d-64b90352396d-host-slash\") pod \"iptables-alerter-bw7xm\" (UID: \"195c1605-870d-4b90-b45d-64b90352396d\") " pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-device-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-kubernetes\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.084996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-systemd\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085553 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.085019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-sys\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085553 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.085045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7wvk\" (UniqueName: \"kubernetes.io/projected/8dd162ac-a8b4-4ed7-9537-e37504a11570-kube-api-access-f7wvk\") pod \"node-ca-qkksk\" (UID: \"8dd162ac-a8b4-4ed7-9537-e37504a11570\") " pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.085553 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.085077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-modprobe-d\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085553 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.085103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-run\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085553 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.085134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-socket-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.085553 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.085157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba498c7-6988-4f37-bc5b-3668877fde57-tmp\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085553 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.085180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-host\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085553 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.085203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-sysctl-d\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.085553 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.085227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-var-lib-kubelet\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.091984 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.091958    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:10:36.092525 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.092505    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:10:36.093058 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.093037    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kxzfz\""
Jul 28 17:10:36.093324 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.093304    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:10:36.093395 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.093371    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:10:36.093540 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.093306    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:10:36.093823 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.093697    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:10:36.096602 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.094330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.096602 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.094402    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:36.096783 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.096735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:36.097036 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.096973    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:36.099385 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.099367    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9sw7f\""
Jul 28 17:10:36.099742 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.099716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.100592 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.100571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:10:36.100794 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.100771    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:10:36.102206 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.102183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.104065 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.102922    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:10:36.104065 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.103150    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:10:36.104065 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.103400    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:10:36.104065 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.103667    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:10:36.104065 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.103858    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:10:36.104065 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.103893    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-skqtd\""
Jul 28 17:10:36.105603 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.105344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:36.105603 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.105411    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:36.105603 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.105491    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:10:36.105816 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.105608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nvqpg\""
Jul 28 17:10:36.123808 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.123785    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2qmqw"
Jul 28 17:10:36.133950 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.133929    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2qmqw"
Jul 28 17:10:36.173955 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.173932    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:10:36.186184 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-cni-netd\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.186300 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/220d6f62-b2f9-47be-bc2b-de661355bc74-ovnkube-script-lib\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.186300 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-var-lib-kubelet\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.186300 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-modprobe-d\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.186517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eca38d07-90a1-4d69-aa43-253bd77a26c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.186517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-etc-kubernetes\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.186517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmxcn\" (UniqueName: \"kubernetes.io/projected/20e8d68a-03b6-46f5-8500-1fd50954ac27-kube-api-access-gmxcn\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:36.186517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-run\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.186517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-modprobe-d\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.186517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-run-ovn\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-log-socket\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-cni-bin\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90388c56-3aae-4240-80a7-ed5560dcde24-agent-certs\") pod \"konnectivity-agent-bfd75\" (UID: \"90388c56-3aae-4240-80a7-ed5560dcde24\") " pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-run-multus-certs\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-run\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-socket-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-run-openvswitch\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1684222-25b4-4a1c-b62f-df4ea459ffe9-kubelet-config\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-socket-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.186831 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-sysctl-d\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-var-lib-kubelet\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-node-log\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-system-cni-dir\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-run-netns\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-sysctl-d\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.186974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eba498c7-6988-4f37-bc5b-3668877fde57-etc-tuned\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8rsr\" (UniqueName: \"kubernetes.io/projected/220d6f62-b2f9-47be-bc2b-de661355bc74-kube-api-access-c8rsr\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-var-lib-kubelet\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-system-cni-dir\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-etc-selinux\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-var-lib-openvswitch\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-run-ovn-kubernetes\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-os-release\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187178    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-etc-selinux\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1684222-25b4-4a1c-b62f-df4ea459ffe9-dbus\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.187292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8dd162ac-a8b4-4ed7-9537-e37504a11570-serviceca\") pod \"node-ca-qkksk\" (UID: \"8dd162ac-a8b4-4ed7-9537-e37504a11570\") " pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7v4r\" (UniqueName: \"kubernetes.io/projected/195c1605-870d-4b90-b45d-64b90352396d-kube-api-access-q7v4r\") pod \"iptables-alerter-bw7xm\" (UID: \"195c1605-870d-4b90-b45d-64b90352396d\") " pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-sysconfig\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkcsx\" (UniqueName: \"kubernetes.io/projected/eba498c7-6988-4f37-bc5b-3668877fde57-kube-api-access-xkcsx\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-etc-openvswitch\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187475    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-cnibin\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-cni-dir\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-sysconfig\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8dd162ac-a8b4-4ed7-9537-e37504a11570-serviceca\") pod \"node-ca-qkksk\" (UID: \"8dd162ac-a8b4-4ed7-9537-e37504a11570\") " pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.187990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-socket-dir-parent\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195c1605-870d-4b90-b45d-64b90352396d-host-slash\") pod \"iptables-alerter-bw7xm\" (UID: \"195c1605-870d-4b90-b45d-64b90352396d\") " pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.188039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-kubernetes\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/195c1605-870d-4b90-b45d-64b90352396d-host-slash\") pod \"iptables-alerter-bw7xm\" (UID: \"195c1605-870d-4b90-b45d-64b90352396d\") " pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-kubernetes\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-sys\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-os-release\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-daemon-config\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqpx\" (UniqueName: \"kubernetes.io/projected/b1da3678-dfa2-4009-8d40-f74dc4c40c68-kube-api-access-7qqpx\") pod \"node-resolver-7tk7q\" (UID: \"b1da3678-dfa2-4009-8d40-f74dc4c40c68\") " pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wvk\" (UniqueName: \"kubernetes.io/projected/8dd162ac-a8b4-4ed7-9537-e37504a11570-kube-api-access-f7wvk\") pod \"node-ca-qkksk\" (UID: \"8dd162ac-a8b4-4ed7-9537-e37504a11570\") " pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-kubelet\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-run-systemd\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7txp\" (UniqueName: \"kubernetes.io/projected/9873ffe2-365c-443c-bdb4-582312b3cd12-kube-api-access-r7txp\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-run-k8s-cni-cncf-io\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba498c7-6988-4f37-bc5b-3668877fde57-tmp\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-systemd-units\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.188686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-run-netns\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90388c56-3aae-4240-80a7-ed5560dcde24-konnectivity-ca\") pod \"konnectivity-agent-bfd75\" (UID: \"90388c56-3aae-4240-80a7-ed5560dcde24\") " pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-cnibin\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-var-lib-cni-bin\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-var-lib-cni-multus\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-conf-dir\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-host\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eca38d07-90a1-4d69-aa43-253bd77a26c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-sys\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9873ffe2-365c-443c-bdb4-582312b3cd12-cni-binary-copy\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1da3678-dfa2-4009-8d40-f74dc4c40c68-hosts-file\") pod \"node-resolver-7tk7q\" (UID: \"b1da3678-dfa2-4009-8d40-f74dc4c40c68\") " pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.188937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/195c1605-870d-4b90-b45d-64b90352396d-iptables-alerter-script\") pod \"iptables-alerter-bw7xm\" (UID: \"195c1605-870d-4b90-b45d-64b90352396d\") " pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/220d6f62-b2f9-47be-bc2b-de661355bc74-env-overrides\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/220d6f62-b2f9-47be-bc2b-de661355bc74-ovn-node-metrics-cert\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eca38d07-90a1-4d69-aa43-253bd77a26c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dd162ac-a8b4-4ed7-9537-e37504a11570-host\") pod \"node-ca-qkksk\" (UID: \"8dd162ac-a8b4-4ed7-9537-e37504a11570\") " pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.189423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-registration-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mdh\" (UniqueName: \"kubernetes.io/projected/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-kube-api-access-z5mdh\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-hostroot\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-sys-fs\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-sysctl-conf\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-lib-modules\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/220d6f62-b2f9-47be-bc2b-de661355bc74-ovnkube-config\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stj9w\" (UniqueName: \"kubernetes.io/projected/eca38d07-90a1-4d69-aa43-253bd77a26c6-kube-api-access-stj9w\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1da3678-dfa2-4009-8d40-f74dc4c40c68-tmp-dir\") pod \"node-resolver-7tk7q\" (UID: \"b1da3678-dfa2-4009-8d40-f74dc4c40c68\") " pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-device-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-systemd\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-slash\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-host\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.189970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-registration-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.190014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-sysctl-conf\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.190055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dd162ac-a8b4-4ed7-9537-e37504a11570-host\") pod \"node-ca-qkksk\" (UID: \"8dd162ac-a8b4-4ed7-9537-e37504a11570\") " pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.190097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-lib-modules\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.190251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.190103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-device-dir\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.190970 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.190148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eba498c7-6988-4f37-bc5b-3668877fde57-etc-systemd\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.190970 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.190201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-sys-fs\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.190970 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.190475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/195c1605-870d-4b90-b45d-64b90352396d-iptables-alerter-script\") pod \"iptables-alerter-bw7xm\" (UID: \"195c1605-870d-4b90-b45d-64b90352396d\") " pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.192023 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.191977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eba498c7-6988-4f37-bc5b-3668877fde57-etc-tuned\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.192267 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.192243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eba498c7-6988-4f37-bc5b-3668877fde57-tmp\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.199043 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.199001    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:36.199043 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.199028    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:36.199043 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.199042    2579 projected.go:196] Error preparing data for projected volume kube-api-access-49q8t for pod openshift-network-diagnostics/network-check-target-t542q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:36.199245 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.199116    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t podName:8124e945-8779-41d3-86f1-726109a4d6d5 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:36.699093732 +0000 UTC m=+2.934059167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-49q8t" (UniqueName: "kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t") pod "network-check-target-t542q" (UID: "8124e945-8779-41d3-86f1-726109a4d6d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:36.199412 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.199390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7v4r\" (UniqueName: \"kubernetes.io/projected/195c1605-870d-4b90-b45d-64b90352396d-kube-api-access-q7v4r\") pod \"iptables-alerter-bw7xm\" (UID: \"195c1605-870d-4b90-b45d-64b90352396d\") " pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.200159 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.200113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkcsx\" (UniqueName: \"kubernetes.io/projected/eba498c7-6988-4f37-bc5b-3668877fde57-kube-api-access-xkcsx\") pod \"tuned-f5tlf\" (UID: \"eba498c7-6988-4f37-bc5b-3668877fde57\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.200458 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.200419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mdh\" (UniqueName: \"kubernetes.io/projected/fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7-kube-api-access-z5mdh\") pod \"aws-ebs-csi-driver-node-zkwhf\" (UID: \"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.201713 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.201667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7wvk\" (UniqueName: \"kubernetes.io/projected/8dd162ac-a8b4-4ed7-9537-e37504a11570-kube-api-access-f7wvk\") pod \"node-ca-qkksk\" (UID: \"8dd162ac-a8b4-4ed7-9537-e37504a11570\") " pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.283103 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.283012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" event={"ID":"67a91ffb87b056369246a9445ebaa0cf","Type":"ContainerStarted","Data":"8c78668400e34558c9ced2e55575f392b9326ed7b2393bf6bd1bb7f9eb20cdc9"}
Jul 28 17:10:36.284351 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.284307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" event={"ID":"6d68380c15b9e00b00440ef022ddd164","Type":"ContainerStarted","Data":"18947e7a7a0b1ed0e5f55cc41cb4bbdef9d71103655c0ecaed13f4957965d27f"}
Jul 28 17:10:36.290753 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.290725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-var-lib-openvswitch\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.290865 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.290764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-run-ovn-kubernetes\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.290865 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.290785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-os-release\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.290865 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.290806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1684222-25b4-4a1c-b62f-df4ea459ffe9-dbus\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.291039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.290907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-etc-openvswitch\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.290926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-cnibin\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.291039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.290942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-cni-dir\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.291039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.290957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-socket-dir-parent\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.291039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.290996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-os-release\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.291039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-daemon-config\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqpx\" (UniqueName: \"kubernetes.io/projected/b1da3678-dfa2-4009-8d40-f74dc4c40c68-kube-api-access-7qqpx\") pod \"node-resolver-7tk7q\" (UID: \"b1da3678-dfa2-4009-8d40-f74dc4c40c68\") " pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-kubelet\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-run-systemd\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7txp\" (UniqueName: \"kubernetes.io/projected/9873ffe2-365c-443c-bdb4-582312b3cd12-kube-api-access-r7txp\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-run-k8s-cni-cncf-io\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-cnibin\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-systemd-units\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-systemd-units\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-os-release\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-socket-dir-parent\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-run-netns\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-var-lib-openvswitch\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1684222-25b4-4a1c-b62f-df4ea459ffe9-dbus\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-etc-openvswitch\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90388c56-3aae-4240-80a7-ed5560dcde24-konnectivity-ca\") pod \"konnectivity-agent-bfd75\" (UID: \"90388c56-3aae-4240-80a7-ed5560dcde24\") " pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:36.291391 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-run-k8s-cni-cncf-io\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-run-netns\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-os-release\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-run-systemd\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-cni-dir\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-run-ovn-kubernetes\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-kubelet\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-cnibin\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-var-lib-cni-bin\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-var-lib-cni-multus\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-conf-dir\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eca38d07-90a1-4d69-aa43-253bd77a26c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9873ffe2-365c-443c-bdb4-582312b3cd12-cni-binary-copy\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-cnibin\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1da3678-dfa2-4009-8d40-f74dc4c40c68-hosts-file\") pod \"node-resolver-7tk7q\" (UID: \"b1da3678-dfa2-4009-8d40-f74dc4c40c68\") " pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1da3678-dfa2-4009-8d40-f74dc4c40c68-hosts-file\") pod \"node-resolver-7tk7q\" (UID: \"b1da3678-dfa2-4009-8d40-f74dc4c40c68\") " pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/220d6f62-b2f9-47be-bc2b-de661355bc74-env-overrides\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/220d6f62-b2f9-47be-bc2b-de661355bc74-ovn-node-metrics-cert\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.292219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-var-lib-cni-multus\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90388c56-3aae-4240-80a7-ed5560dcde24-konnectivity-ca\") pod \"konnectivity-agent-bfd75\" (UID: \"90388c56-3aae-4240-80a7-ed5560dcde24\") " pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eca38d07-90a1-4d69-aa43-253bd77a26c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-daemon-config\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.291991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-hostroot\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/220d6f62-b2f9-47be-bc2b-de661355bc74-ovnkube-config\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-var-lib-cni-bin\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stj9w\" (UniqueName: \"kubernetes.io/projected/eca38d07-90a1-4d69-aa43-253bd77a26c6-kube-api-access-stj9w\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1da3678-dfa2-4009-8d40-f74dc4c40c68-tmp-dir\") pod \"node-resolver-7tk7q\" (UID: \"b1da3678-dfa2-4009-8d40-f74dc4c40c68\") " pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-slash\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-cni-netd\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/220d6f62-b2f9-47be-bc2b-de661355bc74-ovnkube-script-lib\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-var-lib-kubelet\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eca38d07-90a1-4d69-aa43-253bd77a26c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-etc-kubernetes\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxcn\" (UniqueName: \"kubernetes.io/projected/20e8d68a-03b6-46f5-8500-1fd50954ac27-kube-api-access-gmxcn\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-run-ovn\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-log-socket\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-cni-bin\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90388c56-3aae-4240-80a7-ed5560dcde24-agent-certs\") pod \"konnectivity-agent-bfd75\" (UID: \"90388c56-3aae-4240-80a7-ed5560dcde24\") " pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-multus-conf-dir\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-run-multus-certs\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9873ffe2-365c-443c-bdb4-582312b3cd12-cni-binary-copy\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/220d6f62-b2f9-47be-bc2b-de661355bc74-env-overrides\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-run-multus-certs\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eca38d07-90a1-4d69-aa43-253bd77a26c6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-run-openvswitch\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-run-openvswitch\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-var-lib-kubelet\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1684222-25b4-4a1c-b62f-df4ea459ffe9-kubelet-config\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-run-ovn\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-hostroot\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-node-log\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.293870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-log-socket\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-etc-kubernetes\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-system-cni-dir\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-run-netns\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8rsr\" (UniqueName: \"kubernetes.io/projected/220d6f62-b2f9-47be-bc2b-de661355bc74-kube-api-access-c8rsr\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-system-cni-dir\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.292981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-system-cni-dir\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.293065    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1da3678-dfa2-4009-8d40-f74dc4c40c68-tmp-dir\") pod \"node-resolver-7tk7q\" (UID: \"b1da3678-dfa2-4009-8d40-f74dc4c40c68\") " pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eca38d07-90a1-4d69-aa43-253bd77a26c6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.293127    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs podName:20e8d68a-03b6-46f5-8500-1fd50954ac27 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:36.793109669 +0000 UTC m=+3.028075103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs") pod "network-metrics-daemon-zvdqk" (UID: "20e8d68a-03b6-46f5-8500-1fd50954ac27") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293170    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1684222-25b4-4a1c-b62f-df4ea459ffe9-kubelet-config\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-slash\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-node-log\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-cni-netd\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9873ffe2-365c-443c-bdb4-582312b3cd12-host-run-netns\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.294820 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.293335    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:36.295752 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.293381    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret podName:b1684222-25b4-4a1c-b62f-df4ea459ffe9 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:36.793364733 +0000 UTC m=+3.028330167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret") pod "global-pull-secret-syncer-qf77t" (UID: "b1684222-25b4-4a1c-b62f-df4ea459ffe9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:36.295752 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eca38d07-90a1-4d69-aa43-253bd77a26c6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.295752 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eca38d07-90a1-4d69-aa43-253bd77a26c6-system-cni-dir\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.295752 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/220d6f62-b2f9-47be-bc2b-de661355bc74-host-cni-bin\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.295752 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.293728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/220d6f62-b2f9-47be-bc2b-de661355bc74-ovnkube-script-lib\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.295752 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.294884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/220d6f62-b2f9-47be-bc2b-de661355bc74-ovnkube-config\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.295752 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.295212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/220d6f62-b2f9-47be-bc2b-de661355bc74-ovn-node-metrics-cert\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.296102 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.295966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90388c56-3aae-4240-80a7-ed5560dcde24-agent-certs\") pod \"konnectivity-agent-bfd75\" (UID: \"90388c56-3aae-4240-80a7-ed5560dcde24\") " pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:36.301076 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.301044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7txp\" (UniqueName: \"kubernetes.io/projected/9873ffe2-365c-443c-bdb4-582312b3cd12-kube-api-access-r7txp\") pod \"multus-rrq8b\" (UID: \"9873ffe2-365c-443c-bdb4-582312b3cd12\") " pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.302038 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.301990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxcn\" (UniqueName: \"kubernetes.io/projected/20e8d68a-03b6-46f5-8500-1fd50954ac27-kube-api-access-gmxcn\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:36.305840 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.305765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqpx\" (UniqueName: \"kubernetes.io/projected/b1da3678-dfa2-4009-8d40-f74dc4c40c68-kube-api-access-7qqpx\") pod \"node-resolver-7tk7q\" (UID: \"b1da3678-dfa2-4009-8d40-f74dc4c40c68\") " pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.306191 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.306164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stj9w\" (UniqueName: \"kubernetes.io/projected/eca38d07-90a1-4d69-aa43-253bd77a26c6-kube-api-access-stj9w\") pod \"multus-additional-cni-plugins-9sl5f\" (UID: \"eca38d07-90a1-4d69-aa43-253bd77a26c6\") " pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.306598 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.306574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8rsr\" (UniqueName: \"kubernetes.io/projected/220d6f62-b2f9-47be-bc2b-de661355bc74-kube-api-access-c8rsr\") pod \"ovnkube-node-scdr5\" (UID: \"220d6f62-b2f9-47be-bc2b-de661355bc74\") " pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.380819 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.380793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qkksk"
Jul 28 17:10:36.394537 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:36.394507    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd162ac_a8b4_4ed7_9537_e37504a11570.slice/crio-3893577545e77ac97cd9fb4c8b2f235ecd600f101727624251fcde572b6d26c7 WatchSource:0}: Error finding container 3893577545e77ac97cd9fb4c8b2f235ecd600f101727624251fcde572b6d26c7: Status 404 returned error can't find the container with id 3893577545e77ac97cd9fb4c8b2f235ecd600f101727624251fcde572b6d26c7
Jul 28 17:10:36.398681 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.398658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bw7xm"
Jul 28 17:10:36.409026 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:36.409002    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195c1605_870d_4b90_b45d_64b90352396d.slice/crio-38b9da05c05f43625375a776a84c9e3c3baaace038833da6aa4309b021a48243 WatchSource:0}: Error finding container 38b9da05c05f43625375a776a84c9e3c3baaace038833da6aa4309b021a48243: Status 404 returned error can't find the container with id 38b9da05c05f43625375a776a84c9e3c3baaace038833da6aa4309b021a48243
Jul 28 17:10:36.412496 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.412473    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf"
Jul 28 17:10:36.418059 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.418037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f5tlf"
Jul 28 17:10:36.428417 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.428380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7tk7q"
Jul 28 17:10:36.436276 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.436255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:10:36.444099 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.444072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:36.452414 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.452392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9sl5f"
Jul 28 17:10:36.459444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.459423    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rrq8b"
Jul 28 17:10:36.567521 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.567454    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:36.796765 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.796730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:36.796935 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.796787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:36.796935 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:36.796834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:36.796935 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.796922    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:36.797070 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.796941    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:36.797070 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.796950    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:36.797070 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.796967    2579 projected.go:196] Error preparing data for projected volume kube-api-access-49q8t for pod openshift-network-diagnostics/network-check-target-t542q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:36.797070 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.796941    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:36.797070 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.797004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs podName:20e8d68a-03b6-46f5-8500-1fd50954ac27 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:37.796988865 +0000 UTC m=+4.031954302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs") pod "network-metrics-daemon-zvdqk" (UID: "20e8d68a-03b6-46f5-8500-1fd50954ac27") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:36.797070 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.797024    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t podName:8124e945-8779-41d3-86f1-726109a4d6d5 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:37.797013024 +0000 UTC m=+4.031978460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-49q8t" (UniqueName: "kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t") pod "network-check-target-t542q" (UID: "8124e945-8779-41d3-86f1-726109a4d6d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:36.797070 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:36.797044    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret podName:b1684222-25b4-4a1c-b62f-df4ea459ffe9 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:37.797033209 +0000 UTC m=+4.031998646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret") pod "global-pull-secret-syncer-qf77t" (UID: "b1684222-25b4-4a1c-b62f-df4ea459ffe9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:37.137382 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:37.137337    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:36 +0000 UTC" deadline="2028-01-09 11:15:30.685061455 +0000 UTC"
Jul 28 17:10:37.137382 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:37.137378    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12714h4m53.547688932s"
Jul 28 17:10:37.279513 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:37.279455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:37.279668 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:37.279458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:37.279668 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.279640    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:37.279774 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.279711    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:37.288450 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:37.288376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bw7xm" event={"ID":"195c1605-870d-4b90-b45d-64b90352396d","Type":"ContainerStarted","Data":"38b9da05c05f43625375a776a84c9e3c3baaace038833da6aa4309b021a48243"}
Jul 28 17:10:37.290125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:37.290097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qkksk" event={"ID":"8dd162ac-a8b4-4ed7-9537-e37504a11570","Type":"ContainerStarted","Data":"3893577545e77ac97cd9fb4c8b2f235ecd600f101727624251fcde572b6d26c7"}
Jul 28 17:10:37.620129 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:37.619876    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba498c7_6988_4f37_bc5b_3668877fde57.slice/crio-2df3a34a89ebfaa0676f87047bdb6aa6d1936f2ef6b2234bdd55a7a3e60f5746 WatchSource:0}: Error finding container 2df3a34a89ebfaa0676f87047bdb6aa6d1936f2ef6b2234bdd55a7a3e60f5746: Status 404 returned error can't find the container with id 2df3a34a89ebfaa0676f87047bdb6aa6d1936f2ef6b2234bdd55a7a3e60f5746
Jul 28 17:10:37.626992 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:37.626964    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9873ffe2_365c_443c_bdb4_582312b3cd12.slice/crio-9c7c9fa8b9d00e8605ffd6a659ed05c8a4b35f1db9577c7d7e154449d489a3bc WatchSource:0}: Error finding container 9c7c9fa8b9d00e8605ffd6a659ed05c8a4b35f1db9577c7d7e154449d489a3bc: Status 404 returned error can't find the container with id 9c7c9fa8b9d00e8605ffd6a659ed05c8a4b35f1db9577c7d7e154449d489a3bc
Jul 28 17:10:37.652080 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:37.652043    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220d6f62_b2f9_47be_bc2b_de661355bc74.slice/crio-021bfd1d6fbfc97c97f845997323d710514130c18327b77b2735725c0e98eaf7 WatchSource:0}: Error finding container 021bfd1d6fbfc97c97f845997323d710514130c18327b77b2735725c0e98eaf7: Status 404 returned error can't find the container with id 021bfd1d6fbfc97c97f845997323d710514130c18327b77b2735725c0e98eaf7
Jul 28 17:10:37.655226 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:37.655183    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5f2b73_f7f7_4f6a_b31f_3c38b340a3c7.slice/crio-4302c3c25ca3262b0e956ceea28ca71ef3473316fcb127e41b2fbcad871a5b48 WatchSource:0}: Error finding container 4302c3c25ca3262b0e956ceea28ca71ef3473316fcb127e41b2fbcad871a5b48: Status 404 returned error can't find the container with id 4302c3c25ca3262b0e956ceea28ca71ef3473316fcb127e41b2fbcad871a5b48
Jul 28 17:10:37.657433 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:10:37.657392    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1da3678_dfa2_4009_8d40_f74dc4c40c68.slice/crio-9bac087c8743dd1bbfe27b8cf86b52739b2e6b8da4ac67477eb24cd2b0eefc04 WatchSource:0}: Error finding container 9bac087c8743dd1bbfe27b8cf86b52739b2e6b8da4ac67477eb24cd2b0eefc04: Status 404 returned error can't find the container with id 9bac087c8743dd1bbfe27b8cf86b52739b2e6b8da4ac67477eb24cd2b0eefc04
Jul 28 17:10:37.805120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:37.804641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:37.805120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:37.804916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:37.805120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:37.804987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:37.805120 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.804809    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:37.805450 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.805144    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs podName:20e8d68a-03b6-46f5-8500-1fd50954ac27 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:39.805128661 +0000 UTC m=+6.040094093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs") pod "network-metrics-daemon-zvdqk" (UID: "20e8d68a-03b6-46f5-8500-1fd50954ac27") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:37.805450 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.805210    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:37.805450 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.805239    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret podName:b1684222-25b4-4a1c-b62f-df4ea459ffe9 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:39.805227951 +0000 UTC m=+6.040193383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret") pod "global-pull-secret-syncer-qf77t" (UID: "b1684222-25b4-4a1c-b62f-df4ea459ffe9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:37.805656 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.805555    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:37.805656 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.805571    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:37.805656 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.805582    2579 projected.go:196] Error preparing data for projected volume kube-api-access-49q8t for pod openshift-network-diagnostics/network-check-target-t542q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:37.805656 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:37.805617    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t podName:8124e945-8779-41d3-86f1-726109a4d6d5 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:39.805604662 +0000 UTC m=+6.040570094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-49q8t" (UniqueName: "kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t") pod "network-check-target-t542q" (UID: "8124e945-8779-41d3-86f1-726109a4d6d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:38.138463 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.138355    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:36 +0000 UTC" deadline="2028-04-04 20:15:44.808794398 +0000 UTC"
Jul 28 17:10:38.138463 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.138420    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14787h5m6.670393069s"
Jul 28 17:10:38.279693 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.279665    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:38.279867 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:38.279795    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:38.294281 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.294242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" event={"ID":"6d68380c15b9e00b00440ef022ddd164","Type":"ContainerStarted","Data":"510f709f1c80fa144bc6adbb56fe129795f57a63a36504c0451567bb29a8fa36"}
Jul 28 17:10:38.295920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.295887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" event={"ID":"67a91ffb87b056369246a9445ebaa0cf","Type":"ContainerStarted","Data":"74eb33e1c2a18a7e43f9ee22011a4fe779ce3fa62f5f9b0346b2f75b99ef541c"}
Jul 28 17:10:38.297254 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.297228    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf" event={"ID":"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7","Type":"ContainerStarted","Data":"4302c3c25ca3262b0e956ceea28ca71ef3473316fcb127e41b2fbcad871a5b48"}
Jul 28 17:10:38.298521 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.298492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bfd75" event={"ID":"90388c56-3aae-4240-80a7-ed5560dcde24","Type":"ContainerStarted","Data":"ecc321a0322e2ae52d519a7c9d02548ef4a04032a64431c4c221cad07e5ae5dc"}
Jul 28 17:10:38.299778 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.299745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f5tlf" event={"ID":"eba498c7-6988-4f37-bc5b-3668877fde57","Type":"ContainerStarted","Data":"2df3a34a89ebfaa0676f87047bdb6aa6d1936f2ef6b2234bdd55a7a3e60f5746"}
Jul 28 17:10:38.301093 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.301071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" event={"ID":"220d6f62-b2f9-47be-bc2b-de661355bc74","Type":"ContainerStarted","Data":"021bfd1d6fbfc97c97f845997323d710514130c18327b77b2735725c0e98eaf7"}
Jul 28 17:10:38.302325 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.302296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7tk7q" event={"ID":"b1da3678-dfa2-4009-8d40-f74dc4c40c68","Type":"ContainerStarted","Data":"9bac087c8743dd1bbfe27b8cf86b52739b2e6b8da4ac67477eb24cd2b0eefc04"}
Jul 28 17:10:38.303535 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.303511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rrq8b" event={"ID":"9873ffe2-365c-443c-bdb4-582312b3cd12","Type":"ContainerStarted","Data":"9c7c9fa8b9d00e8605ffd6a659ed05c8a4b35f1db9577c7d7e154449d489a3bc"}
Jul 28 17:10:38.304576 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.304554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" event={"ID":"eca38d07-90a1-4d69-aa43-253bd77a26c6","Type":"ContainerStarted","Data":"7bc624a33b675caf2178e281804f645524237c467e258cd4d20ceeb6a4acb466"}
Jul 28 17:10:38.311035 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:38.310993    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-238.ec2.internal" podStartSLOduration=2.3109796 podStartE2EDuration="2.3109796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:38.310247251 +0000 UTC m=+4.545212708" watchObservedRunningTime="2026-07-28 17:10:38.3109796 +0000 UTC m=+4.545945056"
Jul 28 17:10:39.278791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:39.278755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:39.279293 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.278904    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:39.279293 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:39.279285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:39.279439 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.279410    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:39.313098 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:39.313064    2579 generic.go:356] "Generic (PLEG): container finished" podID="67a91ffb87b056369246a9445ebaa0cf" containerID="74eb33e1c2a18a7e43f9ee22011a4fe779ce3fa62f5f9b0346b2f75b99ef541c" exitCode=0
Jul 28 17:10:39.313275 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:39.313174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" event={"ID":"67a91ffb87b056369246a9445ebaa0cf","Type":"ContainerDied","Data":"74eb33e1c2a18a7e43f9ee22011a4fe779ce3fa62f5f9b0346b2f75b99ef541c"}
Jul 28 17:10:39.821331 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:39.821297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:39.821533 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:39.821365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:39.821533 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:39.821397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:39.821533 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.821495    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:39.821533 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.821503    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:39.821533 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.821520    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:39.821533 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.821534    2579 projected.go:196] Error preparing data for projected volume kube-api-access-49q8t for pod openshift-network-diagnostics/network-check-target-t542q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:39.821875 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.821565    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret podName:b1684222-25b4-4a1c-b62f-df4ea459ffe9 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:43.821545395 +0000 UTC m=+10.056510830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret") pod "global-pull-secret-syncer-qf77t" (UID: "b1684222-25b4-4a1c-b62f-df4ea459ffe9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:39.821875 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.821597    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t podName:8124e945-8779-41d3-86f1-726109a4d6d5 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:43.821585403 +0000 UTC m=+10.056550851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-49q8t" (UniqueName: "kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t") pod "network-check-target-t542q" (UID: "8124e945-8779-41d3-86f1-726109a4d6d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:39.821875 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.821658    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:39.821875 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:39.821692    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs podName:20e8d68a-03b6-46f5-8500-1fd50954ac27 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:43.821680383 +0000 UTC m=+10.056645821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs") pod "network-metrics-daemon-zvdqk" (UID: "20e8d68a-03b6-46f5-8500-1fd50954ac27") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:40.282066 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:40.281520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:40.282066 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:40.281650    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:41.280105 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:41.279431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:41.280105 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:41.279557    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:41.280105 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:41.279959    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:41.280105 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:41.280062    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:42.278991 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:42.278962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:42.279532 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:42.279100    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:43.279497 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:43.279469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:43.279878 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.279652    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:43.279878 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:43.279805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:43.279997 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.279932    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:43.854251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:43.854219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:43.854387 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:43.854287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:43.854387 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:43.854318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:43.854515 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.854427    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:43.854515 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.854433    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:43.854515 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.854447    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:43.854515 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.854459    2579 projected.go:196] Error preparing data for projected volume kube-api-access-49q8t for pod openshift-network-diagnostics/network-check-target-t542q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:43.854515 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.854477    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret podName:b1684222-25b4-4a1c-b62f-df4ea459ffe9 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:51.854460264 +0000 UTC m=+18.089425708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret") pod "global-pull-secret-syncer-qf77t" (UID: "b1684222-25b4-4a1c-b62f-df4ea459ffe9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:43.854515 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.854494    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t podName:8124e945-8779-41d3-86f1-726109a4d6d5 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:51.85448368 +0000 UTC m=+18.089449111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-49q8t" (UniqueName: "kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t") pod "network-check-target-t542q" (UID: "8124e945-8779-41d3-86f1-726109a4d6d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:43.854819 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.854578    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:43.854819 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:43.854619    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs podName:20e8d68a-03b6-46f5-8500-1fd50954ac27 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:51.854605151 +0000 UTC m=+18.089570582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs") pod "network-metrics-daemon-zvdqk" (UID: "20e8d68a-03b6-46f5-8500-1fd50954ac27") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:44.283233 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.282792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:44.283233 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:44.282907    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:44.327137 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.327105    2579 generic.go:356] "Generic (PLEG): container finished" podID="eca38d07-90a1-4d69-aa43-253bd77a26c6" containerID="be65597455d330c1a4a3165b8a7bd56c9f211ef08c54eaefc9a41ada064516ea" exitCode=0
Jul 28 17:10:44.327269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.327190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" event={"ID":"eca38d07-90a1-4d69-aa43-253bd77a26c6","Type":"ContainerDied","Data":"be65597455d330c1a4a3165b8a7bd56c9f211ef08c54eaefc9a41ada064516ea"}
Jul 28 17:10:44.330311 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.330251    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" event={"ID":"67a91ffb87b056369246a9445ebaa0cf","Type":"ContainerStarted","Data":"707ffc1ba3be2710730064b387ae46a1e7ff65ebdc4905cd365bc6b8c04083c6"}
Jul 28 17:10:44.333114 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.333089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qkksk" event={"ID":"8dd162ac-a8b4-4ed7-9537-e37504a11570","Type":"ContainerStarted","Data":"babd0e12dc68ec70c54880d03d5b815b7164e99714ebfc34b9ebbbbe5f66a607"}
Jul 28 17:10:44.335897 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.335834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf" event={"ID":"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7","Type":"ContainerStarted","Data":"84fb86a2210594b5ba8bba1154dab4e3da29c5f5833cd67c4d9d1f76929dbda7"}
Jul 28 17:10:44.337810 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.337762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bfd75" event={"ID":"90388c56-3aae-4240-80a7-ed5560dcde24","Type":"ContainerStarted","Data":"2ab92bcc5626fd92399ac5389fbbd4e4b3aa6784b1b610367ac4f5c1fc81fd55"}
Jul 28 17:10:44.340060 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.340038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7tk7q" event={"ID":"b1da3678-dfa2-4009-8d40-f74dc4c40c68","Type":"ContainerStarted","Data":"5a835d6b5bfea8d85d2ecb1853cc5762cdb8d03b574dd2527fcccfedf8f4bad9"}
Jul 28 17:10:44.364903 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.364652    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bfd75" podStartSLOduration=4.39398527 podStartE2EDuration="10.364632154s" totalImagesPullingTime="5.970646884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:44.364136839 +0000 UTC m=+10.599102292" watchObservedRunningTime="2026-07-28 17:10:44.364632154 +0000 UTC m=+10.599597608"
Jul 28 17:10:44.418431 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.418363    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7tk7q" podStartSLOduration=4.464570347 podStartE2EDuration="10.41834285s" totalImagesPullingTime="5.953772503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:44.418152269 +0000 UTC m=+10.653117720" watchObservedRunningTime="2026-07-28 17:10:44.41834285 +0000 UTC m=+10.653308303"
Jul 28 17:10:44.418555 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.418479    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qkksk" podStartSLOduration=3.207370877 podStartE2EDuration="10.418470904s" totalImagesPullingTime="7.211100027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:44.380184018 +0000 UTC m=+10.615149471" watchObservedRunningTime="2026-07-28 17:10:44.418470904 +0000 UTC m=+10.653436357"
Jul 28 17:10:44.447173 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.447127    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-238.ec2.internal" podStartSLOduration=8.447112473 podStartE2EDuration="8.447112473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:44.446535923 +0000 UTC m=+10.681501376" watchObservedRunningTime="2026-07-28 17:10:44.447112473 +0000 UTC m=+10.682077926"
Jul 28 17:10:44.764048 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:44.764023    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:10:45.142294 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:45.142187    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:10:44.764045115Z","UUID":"ff079684-b2c1-4902-be3a-487741b8db87","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:10:45.150186 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:45.149723    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:10:45.150186 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:45.149769    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:10:45.279190 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:45.279160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:45.279360 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:45.279276    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:45.279674 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:45.279638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:45.279761 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:45.279726    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:45.346496 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:45.346259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf" event={"ID":"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7","Type":"ContainerStarted","Data":"d076fcb7c25a75b89edfe845b9903108f859907853fba33dc3bafb5e3a40e36c"}
Jul 28 17:10:45.349871 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:45.349848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bw7xm" event={"ID":"195c1605-870d-4b90-b45d-64b90352396d","Type":"ContainerStarted","Data":"add3c8049a3be7d186a94ff8bbc6884a2cf28fc5ecd27b01e0e306dfbb04fc90"}
Jul 28 17:10:45.372618 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:45.372570    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bw7xm" podStartSLOduration=4.175000743 podStartE2EDuration="11.372551072s" totalImagesPullingTime="7.197550329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:45.372178616 +0000 UTC m=+11.607144068" watchObservedRunningTime="2026-07-28 17:10:45.372551072 +0000 UTC m=+11.607516520"
Jul 28 17:10:46.278769 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:46.278738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:46.278946 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:46.278881    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:46.355322 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:46.355285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf" event={"ID":"fc5f2b73-f7f7-4f6a-b31f-3c38b340a3c7","Type":"ContainerStarted","Data":"3fd9081e0f34e28144d9640a2f17894793adef0b1ababf854f2dd989d00f46fa"}
Jul 28 17:10:47.279010 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:47.278964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:47.279010 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:47.279025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:47.279259 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:47.279143    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:47.279259 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:47.279223    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:47.604706 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:47.604626    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:47.605297 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:47.605279    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:10:47.622176 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:47.622135    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zkwhf" podStartSLOduration=5.891390339 podStartE2EDuration="13.622121447s" totalImagesPullingTime="7.730731108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:46.379205083 +0000 UTC m=+12.614170536" watchObservedRunningTime="2026-07-28 17:10:47.622121447 +0000 UTC m=+13.857086964"
Jul 28 17:10:48.279049 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:48.279020    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:48.279240 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:48.279156    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:48.581375 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:48.581285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:48.682795 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:48.682762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:48.784199 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:48.784154    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:49.279168 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:49.279136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:49.279321 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:49.279228    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:49.279321 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:49.279290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:49.279459 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:49.279428    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:50.278709 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:50.278678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:50.279288 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:50.278779    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:50.717981 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:50.717910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:50.818537 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:50.818505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:51.278974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:51.278945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:51.279417 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:51.278948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:51.279417 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.279074    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:51.279417 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.279156    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:51.414802 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:51.414768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:51.499381 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:51.499330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:51.919707 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:51.919670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:51.919892 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:51.919735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:51.919892 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:51.919769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:51.919892 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.919831    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:51.919892 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.919846    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:51.919892 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.919861    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:51.919892 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.919872    2579 projected.go:196] Error preparing data for projected volume kube-api-access-49q8t for pod openshift-network-diagnostics/network-check-target-t542q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:51.919892 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.919878    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:51.920144 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.919901    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs podName:20e8d68a-03b6-46f5-8500-1fd50954ac27 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:07.919882459 +0000 UTC m=+34.154847892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs") pod "network-metrics-daemon-zvdqk" (UID: "20e8d68a-03b6-46f5-8500-1fd50954ac27") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:51.920144 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.919921    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret podName:b1684222-25b4-4a1c-b62f-df4ea459ffe9 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:07.919910026 +0000 UTC m=+34.154875459 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret") pod "global-pull-secret-syncer-qf77t" (UID: "b1684222-25b4-4a1c-b62f-df4ea459ffe9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:51.920144 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:51.919935    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t podName:8124e945-8779-41d3-86f1-726109a4d6d5 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:07.919928489 +0000 UTC m=+34.154893919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-49q8t" (UniqueName: "kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t") pod "network-check-target-t542q" (UID: "8124e945-8779-41d3-86f1-726109a4d6d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:52.279505 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:52.279422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:52.280083 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:52.279560    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:53.279470 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:53.279436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:53.279630 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:53.279565    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:53.279630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:53.279618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:53.280012 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:53.279713    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:54.282619 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:54.282591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:54.283070 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:54.282710    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:55.279097 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:55.279062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:55.279260 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:55.279105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:55.279260 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:55.279193    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:55.279369 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:55.279344    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:56.279592 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:56.279466    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:56.279923 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:56.279676    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:56.374908 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:56.374879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" event={"ID":"220d6f62-b2f9-47be-bc2b-de661355bc74","Type":"ContainerStarted","Data":"dc8b5160e5ed24a51d2475da2f055e2901c1341706da00d21494ae03fd441b22"}
Jul 28 17:10:56.374988 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:56.374953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" event={"ID":"220d6f62-b2f9-47be-bc2b-de661355bc74","Type":"ContainerStarted","Data":"78e1d2a145438aa479bb64ed8fbf828ae6423cfd0adb5f0b49f20e6122d8c988"}
Jul 28 17:10:56.376456 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:56.376432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rrq8b" event={"ID":"9873ffe2-365c-443c-bdb4-582312b3cd12","Type":"ContainerStarted","Data":"fff0e32b1d371cfe1721887b3f52f903e89450e965f9f3837489e9e7a6ec602f"}
Jul 28 17:10:56.378210 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:56.378188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" event={"ID":"eca38d07-90a1-4d69-aa43-253bd77a26c6","Type":"ContainerStarted","Data":"ae25735d8666d1fd21d8160dda10c176f007f8d763423f0be5e11901487d458c"}
Jul 28 17:10:56.394777 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:56.394725    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rrq8b" podStartSLOduration=3.840766624 podStartE2EDuration="22.394704981s" totalImagesPullingTime="18.553938357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:56.394036949 +0000 UTC m=+22.629002402" watchObservedRunningTime="2026-07-28 17:10:56.394704981 +0000 UTC m=+22.629670434"
Jul 28 17:10:56.527628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:56.527442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:56.619983 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:56.619959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:56.730019 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:56.729965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:57.278479 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:57.278454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:57.278755 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:57.278547    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:57.278755 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:57.278615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:57.278755 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:57.278721    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:57.382851 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:57.382822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" event={"ID":"220d6f62-b2f9-47be-bc2b-de661355bc74","Type":"ContainerStarted","Data":"8d5dfb14f3065db7b93d858b9bd9628e60d05738b7dc3e8aa368ff37ab71fb3c"}
Jul 28 17:10:57.383284 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:57.382892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" event={"ID":"220d6f62-b2f9-47be-bc2b-de661355bc74","Type":"ContainerStarted","Data":"82da9ba1db99c8dce386b4e74dbb68baa6b8aa61526017bbb0a1a3785cdc0727"}
Jul 28 17:10:57.383284 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:57.382942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" event={"ID":"220d6f62-b2f9-47be-bc2b-de661355bc74","Type":"ContainerStarted","Data":"db377c439ef8c47fc45975c89ca6c9be46270192ce069251b4081de2aaf7154b"}
Jul 28 17:10:57.383284 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:57.382992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" event={"ID":"220d6f62-b2f9-47be-bc2b-de661355bc74","Type":"ContainerStarted","Data":"445157be1d298f34d7359e00412b91db5fc8eff3ac688aeaed87ffb82bd863e4"}
Jul 28 17:10:57.384516 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:57.384495    2579 generic.go:356] "Generic (PLEG): container finished" podID="eca38d07-90a1-4d69-aa43-253bd77a26c6" containerID="ae25735d8666d1fd21d8160dda10c176f007f8d763423f0be5e11901487d458c" exitCode=0
Jul 28 17:10:57.384612 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:57.384562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" event={"ID":"eca38d07-90a1-4d69-aa43-253bd77a26c6","Type":"ContainerDied","Data":"ae25735d8666d1fd21d8160dda10c176f007f8d763423f0be5e11901487d458c"}
Jul 28 17:10:58.279263 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:58.279233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:10:58.279435 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:58.279355    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:10:59.278615 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:59.278584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:10:59.279027 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:59.278672    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:10:59.279027 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:59.278726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:10:59.279027 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:10:59.278818    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:10:59.391501 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:59.391464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" event={"ID":"220d6f62-b2f9-47be-bc2b-de661355bc74","Type":"ContainerStarted","Data":"5ba376a5927148111b417bc4915c485ebdee50c5ba4ddc45ffc5820ff46c31bc"}
Jul 28 17:10:59.393499 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:59.393477    2579 generic.go:356] "Generic (PLEG): container finished" podID="eca38d07-90a1-4d69-aa43-253bd77a26c6" containerID="684d44be3ad1b247111ab6ef0c485dc54ce188a3e48fe32df1e33a84a1267822" exitCode=0
Jul 28 17:10:59.393610 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:10:59.393510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" event={"ID":"eca38d07-90a1-4d69-aa43-253bd77a26c6","Type":"ContainerDied","Data":"684d44be3ad1b247111ab6ef0c485dc54ce188a3e48fe32df1e33a84a1267822"}
Jul 28 17:11:00.278855 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:00.278803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:00.279205 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:00.278897    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:11:00.397187 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:00.397142    2579 generic.go:356] "Generic (PLEG): container finished" podID="eca38d07-90a1-4d69-aa43-253bd77a26c6" containerID="904792cbde90757fd1d1bbaae096f949161e83d42ff12ea0ab42a7e30f4ce639" exitCode=0
Jul 28 17:11:00.397300 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:00.397194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" event={"ID":"eca38d07-90a1-4d69-aa43-253bd77a26c6","Type":"ContainerDied","Data":"904792cbde90757fd1d1bbaae096f949161e83d42ff12ea0ab42a7e30f4ce639"}
Jul 28 17:11:00.619444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:00.619377    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:11:00.619576 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:00.619522    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:11:00.619855 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:00.619837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bfd75"
Jul 28 17:11:00.683111 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:00.683080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:01.278940 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:01.278647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:01.279486 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:01.278661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:01.279486 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:01.279012    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:11:01.279486 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:01.279040    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:11:01.403991 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:01.403959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" event={"ID":"220d6f62-b2f9-47be-bc2b-de661355bc74","Type":"ContainerStarted","Data":"95c00ba573f97d3aa122af9b5732a5339bba0c9385cc889df39e18f3e8f79145"}
Jul 28 17:11:01.404248 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:01.404226    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:11:01.404312 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:01.404302    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:11:01.404361 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:01.404351    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:11:01.422686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:01.422632    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:11:01.422908 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:01.422887    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:11:01.437254 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:01.437214    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5" podStartSLOduration=8.863739151 podStartE2EDuration="27.437200116s" totalImagesPullingTime="18.573460965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:01.437048774 +0000 UTC m=+27.672014229" watchObservedRunningTime="2026-07-28 17:11:01.437200116 +0000 UTC m=+27.672165569"
Jul 28 17:11:02.282355 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:02.281902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:02.282355 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:02.282026    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:11:02.941385 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:02.941351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qf77t"]
Jul 28 17:11:02.941571 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:02.941526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:02.941655 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:02.941631    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:11:02.944705 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:02.944672    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t542q"]
Jul 28 17:11:02.944819 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:02.944773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:02.944896 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:02.944859    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:11:02.945701 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:02.945668    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zvdqk"]
Jul 28 17:11:02.945789 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:02.945777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:02.945908 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:02.945884    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:11:04.280458 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:04.280426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:04.280950 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:04.280541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:04.280950 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:04.280542    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:11:04.280950 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:04.280567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:04.280950 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:04.280649    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:11:04.280950 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:04.280737    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:11:05.415002 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:05.414829    2579 generic.go:356] "Generic (PLEG): container finished" podID="eca38d07-90a1-4d69-aa43-253bd77a26c6" containerID="787f132c71671e058dddd7007812eeb6497284cc9f09a671ecee5f08bd6e0c9d" exitCode=0
Jul 28 17:11:05.415002 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:05.414904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" event={"ID":"eca38d07-90a1-4d69-aa43-253bd77a26c6","Type":"ContainerDied","Data":"787f132c71671e058dddd7007812eeb6497284cc9f09a671ecee5f08bd6e0c9d"}
Jul 28 17:11:06.278592 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:06.278568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:06.278592 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:06.278583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:06.278768 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:06.278583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:06.278768 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:06.278663    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:11:06.278837 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:06.278756    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:11:06.278837 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:06.278794    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:11:06.420137 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:06.420108    2579 generic.go:356] "Generic (PLEG): container finished" podID="eca38d07-90a1-4d69-aa43-253bd77a26c6" containerID="016573a398e3d79d0c0c49d65d1b8f45df991cf104cf6b5486f92e7e5f9a2794" exitCode=0
Jul 28 17:11:06.420559 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:06.420156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" event={"ID":"eca38d07-90a1-4d69-aa43-253bd77a26c6","Type":"ContainerDied","Data":"016573a398e3d79d0c0c49d65d1b8f45df991cf104cf6b5486f92e7e5f9a2794"}
Jul 28 17:11:06.576713 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:06.576685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:06.690014 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:06.689982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.425804 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.425774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" event={"ID":"eca38d07-90a1-4d69-aa43-253bd77a26c6","Type":"ContainerStarted","Data":"041e43094fde5240bb09356eccd6f0624733d3aed5889254ca10bedc00ce72a4"}
Jul 28 17:11:07.469874 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.469829    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9sl5f" podStartSLOduration=9.124956932 podStartE2EDuration="33.469814646s" totalImagesPullingTime="24.344857714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:07.469459547 +0000 UTC m=+33.704424999" watchObservedRunningTime="2026-07-28 17:11:07.469814646 +0000 UTC m=+33.704780098"
Jul 28 17:11:07.528622 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.528585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.593654 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.593623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.654710 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.654635    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.754819 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.754789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.812760 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.812731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.946849 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.946785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:07.946849 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.946824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:07.946849 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:07.946844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:07.947117 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:07.946940    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:11:07.947117 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:07.946942    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:11:07.947117 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:07.946998    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs podName:20e8d68a-03b6-46f5-8500-1fd50954ac27 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:39.946981601 +0000 UTC m=+66.181947033 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs") pod "network-metrics-daemon-zvdqk" (UID: "20e8d68a-03b6-46f5-8500-1fd50954ac27") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:11:07.947117 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:07.947006    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:11:07.947117 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:07.947017    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:11:07.947117 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:07.947031    2579 projected.go:196] Error preparing data for projected volume kube-api-access-49q8t for pod openshift-network-diagnostics/network-check-target-t542q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:11:07.947117 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:07.947060    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret podName:b1684222-25b4-4a1c-b62f-df4ea459ffe9 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:39.947043858 +0000 UTC m=+66.182009290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret") pod "global-pull-secret-syncer-qf77t" (UID: "b1684222-25b4-4a1c-b62f-df4ea459ffe9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:11:07.947117 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:07.947082    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t podName:8124e945-8779-41d3-86f1-726109a4d6d5 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:39.947070419 +0000 UTC m=+66.182035870 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-49q8t" (UniqueName: "kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t") pod "network-check-target-t542q" (UID: "8124e945-8779-41d3-86f1-726109a4d6d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:11:08.177161 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:08.177131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:08.276955 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:08.276879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:08.278793 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:08.278759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:08.278907 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:08.278761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:08.279026 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:08.278764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:08.279350 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:08.279207    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t542q" podUID="8124e945-8779-41d3-86f1-726109a4d6d5"
Jul 28 17:11:08.280301 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:08.279516    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zvdqk" podUID="20e8d68a-03b6-46f5-8500-1fd50954ac27"
Jul 28 17:11:08.280301 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:08.279555    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qf77t" podUID="b1684222-25b4-4a1c-b62f-df4ea459ffe9"
Jul 28 17:11:09.113055 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.113028    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-238.ec2.internal" event="NodeReady"
Jul 28 17:11:09.113596 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.113145    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:11:09.158260 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.158218    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dxmmk"]
Jul 28 17:11:09.176300 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.176270    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67d785cb4c-dhvrh"]
Jul 28 17:11:09.176475 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.176386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dxmmk"
Jul 28 17:11:09.179040 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.179008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:11:09.179040 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.179019    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:11:09.179200 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.179180    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-65jvx\""
Jul 28 17:11:09.179252 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.179236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:11:09.200267 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.200247    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5g4wm"]
Jul 28 17:11:09.200390 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.200363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.202982 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.202959    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:11:09.203074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.202982    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xks97\""
Jul 28 17:11:09.203074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.202988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:11:09.203168 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.203083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:11:09.208834 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.208815    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:11:09.222209 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.222185    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dxmmk"]
Jul 28 17:11:09.222321 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.222220    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5g4wm"]
Jul 28 17:11:09.222321 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.222239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67d785cb4c-dhvrh"]
Jul 28 17:11:09.222321 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.222276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.224646 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.224631    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:11:09.224835 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.224820    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kslsq\""
Jul 28 17:11:09.224912 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.224859    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:11:09.356769 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.356737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6tw4\" (UniqueName: \"kubernetes.io/projected/f1dd19dd-f660-4f6e-b57e-3de8377107b8-kube-api-access-h6tw4\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.356942 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.356774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csb24\" (UniqueName: \"kubernetes.io/projected/0f06086f-a8d1-4386-9075-165e5a7ee8b5-kube-api-access-csb24\") pod \"ingress-canary-dxmmk\" (UID: \"0f06086f-a8d1-4386-9075-165e5a7ee8b5\") " pod="openshift-ingress-canary/ingress-canary-dxmmk"
Jul 28 17:11:09.356942 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.356806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dd19dd-f660-4f6e-b57e-3de8377107b8-tmp-dir\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.356942 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.356882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20744d22-0573-4748-8d9c-75d2761cd691-ca-trust-extracted\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.356942 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.356908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-registry-certificates\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.357088 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.356944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-trusted-ca\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.357088 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.356960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-installation-pull-secrets\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.357088 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.356977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dd19dd-f660-4f6e-b57e-3de8377107b8-config-volume\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.357088 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.356990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.357088 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.357014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f06086f-a8d1-4386-9075-165e5a7ee8b5-cert\") pod \"ingress-canary-dxmmk\" (UID: \"0f06086f-a8d1-4386-9075-165e5a7ee8b5\") " pod="openshift-ingress-canary/ingress-canary-dxmmk"
Jul 28 17:11:09.357088 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.357054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-bound-sa-token\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.357281 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.357090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-image-registry-private-configuration\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.357281 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.357110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqcqp\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-kube-api-access-wqcqp\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.357281 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.357164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.457580 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tw4\" (UniqueName: \"kubernetes.io/projected/f1dd19dd-f660-4f6e-b57e-3de8377107b8-kube-api-access-h6tw4\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.457580 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csb24\" (UniqueName: \"kubernetes.io/projected/0f06086f-a8d1-4386-9075-165e5a7ee8b5-kube-api-access-csb24\") pod \"ingress-canary-dxmmk\" (UID: \"0f06086f-a8d1-4386-9075-165e5a7ee8b5\") " pod="openshift-ingress-canary/ingress-canary-dxmmk"
Jul 28 17:11:09.457580 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dd19dd-f660-4f6e-b57e-3de8377107b8-tmp-dir\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.457882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20744d22-0573-4748-8d9c-75d2761cd691-ca-trust-extracted\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.457882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-registry-certificates\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.457882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-trusted-ca\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.457882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-installation-pull-secrets\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.457882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dd19dd-f660-4f6e-b57e-3de8377107b8-config-volume\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.457882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.457882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.457742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f06086f-a8d1-4386-9075-165e5a7ee8b5-cert\") pod \"ingress-canary-dxmmk\" (UID: \"0f06086f-a8d1-4386-9075-165e5a7ee8b5\") " pod="openshift-ingress-canary/ingress-canary-dxmmk"
Jul 28 17:11:09.458215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.458052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dd19dd-f660-4f6e-b57e-3de8377107b8-tmp-dir\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.458215 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.458068    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:09.458215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.458087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20744d22-0573-4748-8d9c-75d2761cd691-ca-trust-extracted\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.458215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.458130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-bound-sa-token\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.458215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.458181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-image-registry-private-configuration\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.458215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.458209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqcqp\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-kube-api-access-wqcqp\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.458561 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.458249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.458561 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.458310    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls podName:f1dd19dd-f660-4f6e-b57e-3de8377107b8 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:09.958292751 +0000 UTC m=+36.193258185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls") pod "dns-default-5g4wm" (UID: "f1dd19dd-f660-4f6e-b57e-3de8377107b8") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:09.458561 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.458352    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:09.458561 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.458364    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d785cb4c-dhvrh: secret "image-registry-tls" not found
Jul 28 17:11:09.458561 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.458437    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls podName:20744d22-0573-4748-8d9c-75d2761cd691 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:09.95842282 +0000 UTC m=+36.193388255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls") pod "image-registry-67d785cb4c-dhvrh" (UID: "20744d22-0573-4748-8d9c-75d2761cd691") : secret "image-registry-tls" not found
Jul 28 17:11:09.458561 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.458429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1dd19dd-f660-4f6e-b57e-3de8377107b8-config-volume\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.458829 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.458630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-registry-certificates\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.458913 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.458888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-trusted-ca\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.461953 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.461925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-installation-pull-secrets\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.462045 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.461939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-image-registry-private-configuration\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.462045 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.462001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f06086f-a8d1-4386-9075-165e5a7ee8b5-cert\") pod \"ingress-canary-dxmmk\" (UID: \"0f06086f-a8d1-4386-9075-165e5a7ee8b5\") " pod="openshift-ingress-canary/ingress-canary-dxmmk"
Jul 28 17:11:09.467258 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.467212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csb24\" (UniqueName: \"kubernetes.io/projected/0f06086f-a8d1-4386-9075-165e5a7ee8b5-kube-api-access-csb24\") pod \"ingress-canary-dxmmk\" (UID: \"0f06086f-a8d1-4386-9075-165e5a7ee8b5\") " pod="openshift-ingress-canary/ingress-canary-dxmmk"
Jul 28 17:11:09.467882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.467796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6tw4\" (UniqueName: \"kubernetes.io/projected/f1dd19dd-f660-4f6e-b57e-3de8377107b8-kube-api-access-h6tw4\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.467882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.467868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqcqp\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-kube-api-access-wqcqp\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.468763 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.468706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-bound-sa-token\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.485857 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.485837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dxmmk"
Jul 28 17:11:09.688298 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.688270    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dxmmk"]
Jul 28 17:11:09.698908 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:09.698885    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f06086f_a8d1_4386_9075_165e5a7ee8b5.slice/crio-c08d63c1b93a5fc82eddbf41c60716af2cf6e6c27e11735a8875792ea0a6a155 WatchSource:0}: Error finding container c08d63c1b93a5fc82eddbf41c60716af2cf6e6c27e11735a8875792ea0a6a155: Status 404 returned error can't find the container with id c08d63c1b93a5fc82eddbf41c60716af2cf6e6c27e11735a8875792ea0a6a155
Jul 28 17:11:09.961255 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.961228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:09.961452 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:09.961305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:09.961452 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.961380    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:09.961452 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.961413    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d785cb4c-dhvrh: secret "image-registry-tls" not found
Jul 28 17:11:09.961452 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.961441    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:09.961612 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.961479    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls podName:20744d22-0573-4748-8d9c-75d2761cd691 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:10.961462877 +0000 UTC m=+37.196428308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls") pod "image-registry-67d785cb4c-dhvrh" (UID: "20744d22-0573-4748-8d9c-75d2761cd691") : secret "image-registry-tls" not found
Jul 28 17:11:09.961612 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:09.961491    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls podName:f1dd19dd-f660-4f6e-b57e-3de8377107b8 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:10.961486173 +0000 UTC m=+37.196451603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls") pod "dns-default-5g4wm" (UID: "f1dd19dd-f660-4f6e-b57e-3de8377107b8") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:10.278830 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.278750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:10.278830 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.278764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:10.278830 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.278777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:10.282948 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.282928    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c6kz\""
Jul 28 17:11:10.283079 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.282991    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5l79z\""
Jul 28 17:11:10.283186 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.282929    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:11:10.283186 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.283167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:11:10.283319 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.283205    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:11:10.283319 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.283211    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:11:10.434625 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.434593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dxmmk" event={"ID":"0f06086f-a8d1-4386-9075-165e5a7ee8b5","Type":"ContainerStarted","Data":"c08d63c1b93a5fc82eddbf41c60716af2cf6e6c27e11735a8875792ea0a6a155"}
Jul 28 17:11:10.738682 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.738578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:10.814157 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.814115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:10.971763 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.971732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:10.971940 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:10.971814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:10.971940 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:10.971912    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:10.971940 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:10.971932    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d785cb4c-dhvrh: secret "image-registry-tls" not found
Jul 28 17:11:10.972089 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:10.971968    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:10.972089 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:10.972000    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls podName:20744d22-0573-4748-8d9c-75d2761cd691 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:12.971980366 +0000 UTC m=+39.206945811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls") pod "image-registry-67d785cb4c-dhvrh" (UID: "20744d22-0573-4748-8d9c-75d2761cd691") : secret "image-registry-tls" not found
Jul 28 17:11:10.972089 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:10.972034    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls podName:f1dd19dd-f660-4f6e-b57e-3de8377107b8 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:12.972017515 +0000 UTC m=+39.206982957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls") pod "dns-default-5g4wm" (UID: "f1dd19dd-f660-4f6e-b57e-3de8377107b8") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:12.440756 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:12.440720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dxmmk" event={"ID":"0f06086f-a8d1-4386-9075-165e5a7ee8b5","Type":"ContainerStarted","Data":"2d1ece7ed5a5a1e0be45ce7bffd33f96c496d87bb99e2274c33082c281cf273a"}
Jul 28 17:11:12.458047 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:12.458005    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dxmmk" podStartSLOduration=0.881202665 podStartE2EDuration="3.457987771s" totalImagesPullingTime="2.576785106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:12.45678685 +0000 UTC m=+38.691752336" watchObservedRunningTime="2026-07-28 17:11:12.457987771 +0000 UTC m=+38.692953222"
Jul 28 17:11:12.698990 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:12.698917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:12.775286 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:12.775247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:12.987056 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:12.986977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:12.987056 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:12.987050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:12.987273 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:12.987140    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:12.987273 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:12.987155    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:12.987273 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:12.987225    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls podName:f1dd19dd-f660-4f6e-b57e-3de8377107b8 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:16.987195725 +0000 UTC m=+43.222161168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls") pod "dns-default-5g4wm" (UID: "f1dd19dd-f660-4f6e-b57e-3de8377107b8") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:12.987273 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:12.987158    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d785cb4c-dhvrh: secret "image-registry-tls" not found
Jul 28 17:11:12.987498 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:12.987277    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls podName:20744d22-0573-4748-8d9c-75d2761cd691 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:16.987262588 +0000 UTC m=+43.222228018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls") pod "image-registry-67d785cb4c-dhvrh" (UID: "20744d22-0573-4748-8d9c-75d2761cd691") : secret "image-registry-tls" not found
Jul 28 17:11:15.099072 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.099017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.329057 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.329025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.549675 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.549646    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6ffmn"]
Jul 28 17:11:15.580505 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.580473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.583432 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.583394    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6ffmn"]
Jul 28 17:11:15.583525 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.583510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.586060 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.586040    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:11:15.586198 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.586172    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:11:15.586314 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.586180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:11:15.586314 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.586234    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:11:15.586314 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.586252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rd4np\""
Jul 28 17:11:15.674222 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.674037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.706554 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.706525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.706554 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.706556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.706737 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.706587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5f7\" (UniqueName: \"kubernetes.io/projected/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-api-access-zz5f7\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.706737 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.706628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21b940d6-a972-4a7e-b3e2-3054c9bac62f-crio-socket\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.706737 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.706643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21b940d6-a972-4a7e-b3e2-3054c9bac62f-data-volume\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.807337 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.807269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.807337 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.807326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5f7\" (UniqueName: \"kubernetes.io/projected/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-api-access-zz5f7\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.807508 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.807377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21b940d6-a972-4a7e-b3e2-3054c9bac62f-crio-socket\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.807508 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.807396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21b940d6-a972-4a7e-b3e2-3054c9bac62f-data-volume\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.807508 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:15.807487    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:15.807508 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.807502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.807649 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:15.807551    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:16.307530725 +0000 UTC m=+42.542496160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:15.807649 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:15.807581    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:15.807649 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.807620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21b940d6-a972-4a7e-b3e2-3054c9bac62f-crio-socket\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.807756 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:15.807651    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:16.307637981 +0000 UTC m=+42.542603411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:15.807795 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.807755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21b940d6-a972-4a7e-b3e2-3054c9bac62f-data-volume\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.827168 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.827139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5f7\" (UniqueName: \"kubernetes.io/projected/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-api-access-zz5f7\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:15.867950 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.867923    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.962946 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:15.962914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:16.069959 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.069877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:16.125839 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.125812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:16.310304 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.310276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:16.310484 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.310316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:16.310484 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:16.310412    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:16.310484 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:16.310458    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:16.310596 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:16.310495    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:17.310474793 +0000 UTC m=+43.545440232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:16.310596 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:16.310512    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:17.310505008 +0000 UTC m=+43.545470441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:16.346614 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.346535    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:16.452306 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.452272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:16.847449 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.847390    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs"]
Jul 28 17:11:16.890385 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.890351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs"]
Jul 28 17:11:16.890534 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.890469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs"
Jul 28 17:11:16.893415 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.893362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:16.893613 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.893371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tph8h\""
Jul 28 17:11:16.893613 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.893465    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 17:11:16.894057 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.894040    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:16.921778 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.921724    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7tk7q_b1da3678-dfa2-4009-8d40-f74dc4c40c68/dns-node-resolver/0.log"
Jul 28 17:11:16.992739 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:16.992704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:17.015944 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.015915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:17.016103 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.015984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:17.016103 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.016086    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:17.016237 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.016108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdhf\" (UniqueName: \"kubernetes.io/projected/f496a721-c735-4949-b4a7-12bcbdcbecf0-kube-api-access-4pdhf\") pod \"migrator-6865c9967-9t7rs\" (UID: \"f496a721-c735-4949-b4a7-12bcbdcbecf0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs"
Jul 28 17:11:17.016237 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.016090    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:17.016237 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.016158    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls podName:f1dd19dd-f660-4f6e-b57e-3de8377107b8 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:25.016139798 +0000 UTC m=+51.251105230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls") pod "dns-default-5g4wm" (UID: "f1dd19dd-f660-4f6e-b57e-3de8377107b8") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:17.016237 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.016162    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d785cb4c-dhvrh: secret "image-registry-tls" not found
Jul 28 17:11:17.016237 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.016195    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls podName:20744d22-0573-4748-8d9c-75d2761cd691 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:25.016185073 +0000 UTC m=+51.251150517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls") pod "image-registry-67d785cb4c-dhvrh" (UID: "20744d22-0573-4748-8d9c-75d2761cd691") : secret "image-registry-tls" not found
Jul 28 17:11:17.117604 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.117534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdhf\" (UniqueName: \"kubernetes.io/projected/f496a721-c735-4949-b4a7-12bcbdcbecf0-kube-api-access-4pdhf\") pod \"migrator-6865c9967-9t7rs\" (UID: \"f496a721-c735-4949-b4a7-12bcbdcbecf0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs"
Jul 28 17:11:17.125941 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.125915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdhf\" (UniqueName: \"kubernetes.io/projected/f496a721-c735-4949-b4a7-12bcbdcbecf0-kube-api-access-4pdhf\") pod \"migrator-6865c9967-9t7rs\" (UID: \"f496a721-c735-4949-b4a7-12bcbdcbecf0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs"
Jul 28 17:11:17.199135 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.199096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs"
Jul 28 17:11:17.265109 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.265077    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"]
Jul 28 17:11:17.277754 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.277701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:17.278681 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.278658    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"]
Jul 28 17:11:17.280239 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.280217    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-n9vt5\""
Jul 28 17:11:17.281305 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.281278    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 17:11:17.281447 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.281348    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 17:11:17.319359 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.319331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:17.319500 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.319370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:17.319500 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.319479    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:17.319614 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.319500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/628667c1-e4ab-4462-aceb-a83ef4a9c73b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:17.319614 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.319531    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:19.319513863 +0000 UTC m=+45.554479301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:17.319718 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.319648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:17.319770 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.319737    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:17.319812 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.319786    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:19.31977025 +0000 UTC m=+45.554735684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:17.420060 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.419969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/628667c1-e4ab-4462-aceb-a83ef4a9c73b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:17.420456 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.420423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:17.420567 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.420552    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:11:17.420633 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.420623    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert podName:628667c1-e4ab-4462-aceb-a83ef4a9c73b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:17.920603877 +0000 UTC m=+44.155569311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fsxj8" (UID: "628667c1-e4ab-4462-aceb-a83ef4a9c73b") : secret "networking-console-plugin-cert" not found
Jul 28 17:11:17.420677 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.420650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/628667c1-e4ab-4462-aceb-a83ef4a9c73b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:17.924573 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.924536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:17.924762 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.924710    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:11:17.924822 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:17.924791    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert podName:628667c1-e4ab-4462-aceb-a83ef4a9c73b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:18.924769497 +0000 UTC m=+45.159734930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fsxj8" (UID: "628667c1-e4ab-4462-aceb-a83ef4a9c73b") : secret "networking-console-plugin-cert" not found
Jul 28 17:11:17.941453 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.941423    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-kq5vf"]
Jul 28 17:11:17.964750 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.964722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-kq5vf"]
Jul 28 17:11:17.964899 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.964812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:17.967691 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.967670    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 17:11:17.967789 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.967745    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 17:11:17.968565 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.968544    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 17:11:17.968825 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.968805    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 17:11:17.968932 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.968808    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pncr6\""
Jul 28 17:11:17.975377 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:17.975353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:18.025293 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.025104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f-signing-key\") pod \"service-ca-658d7b7c78-kq5vf\" (UID: \"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f\") " pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.025293 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.025199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f-signing-cabundle\") pod \"service-ca-658d7b7c78-kq5vf\" (UID: \"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f\") " pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.025293 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.025231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb94g\" (UniqueName: \"kubernetes.io/projected/2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f-kube-api-access-hb94g\") pod \"service-ca-658d7b7c78-kq5vf\" (UID: \"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f\") " pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.069373 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.069343    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs"]
Jul 28 17:11:18.072482 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:18.072456    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf496a721_c735_4949_b4a7_12bcbdcbecf0.slice/crio-5a9e1a8bd735831d94978651551e5136ac8df985a52dd64b46cd04cb3ff70229 WatchSource:0}: Error finding container 5a9e1a8bd735831d94978651551e5136ac8df985a52dd64b46cd04cb3ff70229: Status 404 returned error can't find the container with id 5a9e1a8bd735831d94978651551e5136ac8df985a52dd64b46cd04cb3ff70229
Jul 28 17:11:18.073112 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.073089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:18.125967 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.125934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f-signing-cabundle\") pod \"service-ca-658d7b7c78-kq5vf\" (UID: \"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f\") " pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.126770 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.125991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb94g\" (UniqueName: \"kubernetes.io/projected/2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f-kube-api-access-hb94g\") pod \"service-ca-658d7b7c78-kq5vf\" (UID: \"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f\") " pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.126770 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.126036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f-signing-key\") pod \"service-ca-658d7b7c78-kq5vf\" (UID: \"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f\") " pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.126770 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.126695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f-signing-cabundle\") pod \"service-ca-658d7b7c78-kq5vf\" (UID: \"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f\") " pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.128578 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.128556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f-signing-key\") pod \"service-ca-658d7b7c78-kq5vf\" (UID: \"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f\") " pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.134331 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.134276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb94g\" (UniqueName: \"kubernetes.io/projected/2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f-kube-api-access-hb94g\") pod \"service-ca-658d7b7c78-kq5vf\" (UID: \"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f\") " pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.273840 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.273759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf"
Jul 28 17:11:18.325521 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.325498    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qkksk_8dd162ac-a8b4-4ed7-9537-e37504a11570/node-ca/0.log"
Jul 28 17:11:18.343168 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.342924    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:18.411177 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.410948    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:18.455670 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.455646    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-kq5vf"]
Jul 28 17:11:18.456670 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.456647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs" event={"ID":"f496a721-c735-4949-b4a7-12bcbdcbecf0","Type":"ContainerStarted","Data":"5a9e1a8bd735831d94978651551e5136ac8df985a52dd64b46cd04cb3ff70229"}
Jul 28 17:11:18.459216 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:18.459190    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0e5239_cadd_4854_8deb_e2e4bc2e3a0f.slice/crio-d9d32754536b17d564915e9d2b4af1eec8137e3ee7c7e9e84a72bd9260655de3 WatchSource:0}: Error finding container d9d32754536b17d564915e9d2b4af1eec8137e3ee7c7e9e84a72bd9260655de3: Status 404 returned error can't find the container with id d9d32754536b17d564915e9d2b4af1eec8137e3ee7c7e9e84a72bd9260655de3
Jul 28 17:11:18.921038 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.920999    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dxmmk_0f06086f-a8d1-4386-9075-165e5a7ee8b5/serve-healthcheck-canary/0.log"
Jul 28 17:11:18.931313 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:18.931284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:18.931467 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:18.931451    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:11:18.931526 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:18.931517    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert podName:628667c1-e4ab-4462-aceb-a83ef4a9c73b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:20.931501727 +0000 UTC m=+47.166467157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fsxj8" (UID: "628667c1-e4ab-4462-aceb-a83ef4a9c73b") : secret "networking-console-plugin-cert" not found
Jul 28 17:11:19.043576 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:19.043539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:19.334989 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:19.334952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:19.335544 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:19.334998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:19.335544 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:19.335094    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:19.335544 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:19.335177    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:23.335155257 +0000 UTC m=+49.570120690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:19.335544 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:19.335193    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:19.335544 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:19.335251    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:23.335235015 +0000 UTC m=+49.570200445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:19.461430 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:19.461377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f5tlf" event={"ID":"eba498c7-6988-4f37-bc5b-3668877fde57","Type":"ContainerStarted","Data":"344d1d796fe22ef08886852eeb05be3a264019b74a4dbd65bf93023432a91e4e"}
Jul 28 17:11:19.462709 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:19.462683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf" event={"ID":"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f","Type":"ContainerStarted","Data":"d9d32754536b17d564915e9d2b4af1eec8137e3ee7c7e9e84a72bd9260655de3"}
Jul 28 17:11:19.479242 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:19.479194    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f5tlf" podStartSLOduration=4.777605991 podStartE2EDuration="45.479176634s" totalImagesPullingTime="40.701570643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:19.478229539 +0000 UTC m=+45.713194991" watchObservedRunningTime="2026-07-28 17:11:19.479176634 +0000 UTC m=+45.714142089"
Jul 28 17:11:20.466559 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:20.466523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs" event={"ID":"f496a721-c735-4949-b4a7-12bcbdcbecf0","Type":"ContainerStarted","Data":"462e6b1d961aeea64000105c0e8b415a12c8e9817ae150761be806188f238084"}
Jul 28 17:11:20.467024 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:20.466579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs" event={"ID":"f496a721-c735-4949-b4a7-12bcbdcbecf0","Type":"ContainerStarted","Data":"cd3b68a3718e46874204c4d4e60edfaa4f3c77d95a3f173f947f31e497ad71f2"}
Jul 28 17:11:20.484787 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:20.484743    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9t7rs" podStartSLOduration=2.929065154 podStartE2EDuration="4.484728004s" totalImagesPullingTime="1.55566285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:20.482881573 +0000 UTC m=+46.717847022" watchObservedRunningTime="2026-07-28 17:11:20.484728004 +0000 UTC m=+46.719693457"
Jul 28 17:11:20.947843 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:20.947809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:20.948013 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:20.947950    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:11:20.948013 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:20.948010    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert podName:628667c1-e4ab-4462-aceb-a83ef4a9c73b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:24.94799599 +0000 UTC m=+51.182961421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fsxj8" (UID: "628667c1-e4ab-4462-aceb-a83ef4a9c73b") : secret "networking-console-plugin-cert" not found
Jul 28 17:11:21.347359 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:21.347323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.425680 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:21.425651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.470052 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:21.470024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf" event={"ID":"2a0e5239-cadd-4854-8deb-e2e4bc2e3a0f","Type":"ContainerStarted","Data":"0de51d1ebb9d0683e7cb837fc51c2874c4d3893ec2290e34fd28984c0e60f6bb"}
Jul 28 17:11:21.488649 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:21.488610    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-kq5vf" podStartSLOduration=2.193032314 podStartE2EDuration="4.488595809s" totalImagesPullingTime="2.295563495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:21.488029059 +0000 UTC m=+47.722994511" watchObservedRunningTime="2026-07-28 17:11:21.488595809 +0000 UTC m=+47.723561262"
Jul 28 17:11:21.699245 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:21.699166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.832705 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:21.832660    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.933359 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:21.933327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:22.034803 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:22.034704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:22.222302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:22.222267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:22.350623 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:22.350584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:22.608889 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:22.608814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:23.190556 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:23.190520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:23.375168 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:23.375136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:23.375303 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:23.375255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:23.375303 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:23.375287    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:23.375386 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:23.375325    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:23.375386 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:23.375354    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:31.375337655 +0000 UTC m=+57.610303084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:23.375386 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:23.375370    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:31.3753634 +0000 UTC m=+57.610328829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:23.792700 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:23.792673    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:23.996087 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:23.996053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:24.189532 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:24.189492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:24.311265 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:24.311226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:24.583096 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:24.583062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:24.988316 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:24.988235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:24.988854 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:24.988420    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 17:11:24.988854 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:24.988501    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert podName:628667c1-e4ab-4462-aceb-a83ef4a9c73b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:32.988479595 +0000 UTC m=+59.223445036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-fsxj8" (UID: "628667c1-e4ab-4462-aceb-a83ef4a9c73b") : secret "networking-console-plugin-cert" not found
Jul 28 17:11:25.089517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.089488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:25.089695 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.089545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:25.089695 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:25.089657    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:25.089695 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:25.089677    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67d785cb4c-dhvrh: secret "image-registry-tls" not found
Jul 28 17:11:25.089878 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:25.089742    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls podName:20744d22-0573-4748-8d9c-75d2761cd691 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:41.089720582 +0000 UTC m=+67.324686012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls") pod "image-registry-67d785cb4c-dhvrh" (UID: "20744d22-0573-4748-8d9c-75d2761cd691") : secret "image-registry-tls" not found
Jul 28 17:11:25.093632 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.093607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1dd19dd-f660-4f6e-b57e-3de8377107b8-metrics-tls\") pod \"dns-default-5g4wm\" (UID: \"f1dd19dd-f660-4f6e-b57e-3de8377107b8\") " pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:25.130576 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.130553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:25.189322 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.188708    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:25.264671 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.264597    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5g4wm"]
Jul 28 17:11:25.267319 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:25.267274    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dd19dd_f660_4f6e_b57e_3de8377107b8.slice/crio-27455bdbd82f13843fa5e7ef28752221074876946b02949f9070fe711f4f93d7 WatchSource:0}: Error finding container 27455bdbd82f13843fa5e7ef28752221074876946b02949f9070fe711f4f93d7: Status 404 returned error can't find the container with id 27455bdbd82f13843fa5e7ef28752221074876946b02949f9070fe711f4f93d7
Jul 28 17:11:25.392696 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.392667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:25.479374 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.479342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5g4wm" event={"ID":"f1dd19dd-f660-4f6e-b57e-3de8377107b8","Type":"ContainerStarted","Data":"27455bdbd82f13843fa5e7ef28752221074876946b02949f9070fe711f4f93d7"}
Jul 28 17:11:25.596978 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.596936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:25.834067 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.834031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:25.990775 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:25.990691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.205453 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:26.205389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.260830 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:26.260709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.585118 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:26.585084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.180251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:27.180210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.487127 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:27.487099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5g4wm" event={"ID":"f1dd19dd-f660-4f6e-b57e-3de8377107b8","Type":"ContainerStarted","Data":"fdeb867ed1d371b8cbb97b65a1b977ac4e86f77cf97a6cb97812d0b52547572b"}
Jul 28 17:11:27.487336 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:27.487148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5g4wm" event={"ID":"f1dd19dd-f660-4f6e-b57e-3de8377107b8","Type":"ContainerStarted","Data":"fb9b80a63d3d8a7cb7ec9f95721ba848120495b22717ea08231080da3709184e"}
Jul 28 17:11:27.487336 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:27.487248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:27.508202 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:27.508164    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5g4wm" podStartSLOduration=16.845221274 podStartE2EDuration="18.508151252s" totalImagesPullingTime="1.662929978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:27.506691161 +0000 UTC m=+53.741656613" watchObservedRunningTime="2026-07-28 17:11:27.508151252 +0000 UTC m=+53.743116704"
Jul 28 17:11:27.784500 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:27.784411    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.977230 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:27.977179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:28.032415 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:28.032371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:28.182974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:28.182933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:28.422847 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:28.422816    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:28.797141 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:28.797105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:29.394695 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:29.394653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:29.594190 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:29.594161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:29.986428 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:29.986380    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:30.189953 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:30.189918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:30.398679 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:30.398646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:30.588276 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:30.588239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:31.189199 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:31.189171    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:31.439920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:31.439844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:31.439920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:31.439879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:31.440286 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:31.440000    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:31.440286 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:31.440047    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls podName:21b940d6-a972-4a7e-b3e2-3054c9bac62f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:47.440032844 +0000 UTC m=+73.674998274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ffmn" (UID: "21b940d6-a972-4a7e-b3e2-3054c9bac62f") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:31.440449 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:31.440432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21b940d6-a972-4a7e-b3e2-3054c9bac62f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:31.588766 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:31.588730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:31.782753 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:31.782665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:32.183027 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:32.182993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:32.284964 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:32.284920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:32.384140 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:32.384109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:32.446633 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:32.446548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:32.569059 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:32.569027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:32.928474 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:32.928441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:33.030555 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.030524    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:33.054520 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.054495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:33.056886 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.056862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/628667c1-e4ab-4462-aceb-a83ef4a9c73b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-fsxj8\" (UID: \"628667c1-e4ab-4462-aceb-a83ef4a9c73b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:33.188538 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.188474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"
Jul 28 17:11:33.308786 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.308756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8"]
Jul 28 17:11:33.311890 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:33.311858    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628667c1_e4ab_4462_aceb_a83ef4a9c73b.slice/crio-3b2284a29f591c03e5a375fca35cf03cc67f04e902a10cdd2b05e3a17a8625af WatchSource:0}: Error finding container 3b2284a29f591c03e5a375fca35cf03cc67f04e902a10cdd2b05e3a17a8625af: Status 404 returned error can't find the container with id 3b2284a29f591c03e5a375fca35cf03cc67f04e902a10cdd2b05e3a17a8625af
Jul 28 17:11:33.327115 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.327085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:33.417318 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.417293    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scdr5"
Jul 28 17:11:33.428963 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.428929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:33.511661 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.511607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8" event={"ID":"628667c1-e4ab-4462-aceb-a83ef4a9c73b","Type":"ContainerStarted","Data":"3b2284a29f591c03e5a375fca35cf03cc67f04e902a10cdd2b05e3a17a8625af"}
Jul 28 17:11:33.566528 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.566503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:33.703529 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.703488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:33.797068 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.797035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:33.947616 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:33.947573    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:34.151048 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:34.150974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:34.377273 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:34.377238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:34.516032 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:34.515944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8" event={"ID":"628667c1-e4ab-4462-aceb-a83ef4a9c73b","Type":"ContainerStarted","Data":"380b31d0f98ff5f5c50e32a3a0ce7c9dd754b3dea8112252cfd16c5ec8e43739"}
Jul 28 17:11:35.375782 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:35.375746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:35.477197 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:35.477162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:37.492924 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:37.492896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5g4wm"
Jul 28 17:11:37.512396 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:37.512348    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-fsxj8" podStartSLOduration=19.639083429 podStartE2EDuration="20.512334804s" totalImagesPullingTime="873.251375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:34.53677815 +0000 UTC m=+60.771743602" watchObservedRunningTime="2026-07-28 17:11:37.512334804 +0000 UTC m=+63.747300256"
Jul 28 17:11:40.001304 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.001255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:40.001735 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.001323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:40.001735 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.001350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:40.004241 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.004221    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:11:40.004932 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.004911    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:11:40.005031 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.004914    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:11:40.014454 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.014429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1684222-25b4-4a1c-b62f-df4ea459ffe9-original-pull-secret\") pod \"global-pull-secret-syncer-qf77t\" (UID: \"b1684222-25b4-4a1c-b62f-df4ea459ffe9\") " pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:40.014538 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.014455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20e8d68a-03b6-46f5-8500-1fd50954ac27-metrics-certs\") pod \"network-metrics-daemon-zvdqk\" (UID: \"20e8d68a-03b6-46f5-8500-1fd50954ac27\") " pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:40.014814 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.014800    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:11:40.024955 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.024931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49q8t\" (UniqueName: \"kubernetes.io/projected/8124e945-8779-41d3-86f1-726109a4d6d5-kube-api-access-49q8t\") pod \"network-check-target-t542q\" (UID: \"8124e945-8779-41d3-86f1-726109a4d6d5\") " pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:40.294390 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.294305    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c6kz\""
Jul 28 17:11:40.299604 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.299585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qf77t"
Jul 28 17:11:40.301710 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.301694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zvdqk"
Jul 28 17:11:40.307508 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.307492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5l79z\""
Jul 28 17:11:40.315615 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.315592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:40.447725 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.447638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zvdqk"]
Jul 28 17:11:40.450321 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:40.450292    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e8d68a_03b6_46f5_8500_1fd50954ac27.slice/crio-3c77fdfbb10b767867881efb314e4978d322021c5e8209c11633f9584bafd35c WatchSource:0}: Error finding container 3c77fdfbb10b767867881efb314e4978d322021c5e8209c11633f9584bafd35c: Status 404 returned error can't find the container with id 3c77fdfbb10b767867881efb314e4978d322021c5e8209c11633f9584bafd35c
Jul 28 17:11:40.467957 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.467936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qf77t"]
Jul 28 17:11:40.471444 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:40.471424    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1684222_25b4_4a1c_b62f_df4ea459ffe9.slice/crio-7e7df98683fcd8bbf44c542ada23d9604c6d71e239cc1d3e1f6f1b2f31aed051 WatchSource:0}: Error finding container 7e7df98683fcd8bbf44c542ada23d9604c6d71e239cc1d3e1f6f1b2f31aed051: Status 404 returned error can't find the container with id 7e7df98683fcd8bbf44c542ada23d9604c6d71e239cc1d3e1f6f1b2f31aed051
Jul 28 17:11:40.481221 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.481201    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t542q"]
Jul 28 17:11:40.483923 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:40.483898    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8124e945_8779_41d3_86f1_726109a4d6d5.slice/crio-806df60ae3d99978c47e2929beaffd0d5e41796bc6a7d040c2e04d227b9f84da WatchSource:0}: Error finding container 806df60ae3d99978c47e2929beaffd0d5e41796bc6a7d040c2e04d227b9f84da: Status 404 returned error can't find the container with id 806df60ae3d99978c47e2929beaffd0d5e41796bc6a7d040c2e04d227b9f84da
Jul 28 17:11:40.534613 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.534587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t542q" event={"ID":"8124e945-8779-41d3-86f1-726109a4d6d5","Type":"ContainerStarted","Data":"806df60ae3d99978c47e2929beaffd0d5e41796bc6a7d040c2e04d227b9f84da"}
Jul 28 17:11:40.535619 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.535596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zvdqk" event={"ID":"20e8d68a-03b6-46f5-8500-1fd50954ac27","Type":"ContainerStarted","Data":"3c77fdfbb10b767867881efb314e4978d322021c5e8209c11633f9584bafd35c"}
Jul 28 17:11:40.536618 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:40.536600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qf77t" event={"ID":"b1684222-25b4-4a1c-b62f-df4ea459ffe9","Type":"ContainerStarted","Data":"7e7df98683fcd8bbf44c542ada23d9604c6d71e239cc1d3e1f6f1b2f31aed051"}
Jul 28 17:11:41.107136 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.107098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:41.110033 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.109987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"image-registry-67d785cb4c-dhvrh\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") " pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:41.312591 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.312555    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xks97\""
Jul 28 17:11:41.320604 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.320361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:41.565921 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.565386    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67d785cb4c-dhvrh"]
Jul 28 17:11:41.568515 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:41.568484    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20744d22_0573_4748_8d9c_75d2761cd691.slice/crio-63ddd9f3d985adc7466a68763ffc3e4588b72caec631cfcbf80b1e6be6c5aef1 WatchSource:0}: Error finding container 63ddd9f3d985adc7466a68763ffc3e4588b72caec631cfcbf80b1e6be6c5aef1: Status 404 returned error can't find the container with id 63ddd9f3d985adc7466a68763ffc3e4588b72caec631cfcbf80b1e6be6c5aef1
Jul 28 17:11:41.587737 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.587631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:41.920395 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.920274    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"]
Jul 28 17:11:41.923657 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.923632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"
Jul 28 17:11:41.930587 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.930562    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:11:41.931933 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.931562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-d7dkv\""
Jul 28 17:11:41.931933 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.931783    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:11:41.931933 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.931794    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 17:11:41.931933 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.931812    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:11:41.947048 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.947025    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"]
Jul 28 17:11:41.976533 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:41.976506    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67d785cb4c-dhvrh"]
Jul 28 17:11:42.014689 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.014627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w589v\" (UniqueName: \"kubernetes.io/projected/af35c088-4a35-47a4-a2a8-d97d9826c925-kube-api-access-w589v\") pod \"managed-serviceaccount-addon-agent-cf69d56-bh7fz\" (UID: \"af35c088-4a35-47a4-a2a8-d97d9826c925\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"
Jul 28 17:11:42.014837 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.014775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af35c088-4a35-47a4-a2a8-d97d9826c925-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cf69d56-bh7fz\" (UID: \"af35c088-4a35-47a4-a2a8-d97d9826c925\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"
Jul 28 17:11:42.026695 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.026468    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-mr525"]
Jul 28 17:11:42.029901 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.029875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-mr525"
Jul 28 17:11:42.033690 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.033669    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:11:42.033793 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.033741    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fnfgt\""
Jul 28 17:11:42.039720 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.039547    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:11:42.071701 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.071651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-mr525"]
Jul 28 17:11:42.115385 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.115313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w589v\" (UniqueName: \"kubernetes.io/projected/af35c088-4a35-47a4-a2a8-d97d9826c925-kube-api-access-w589v\") pod \"managed-serviceaccount-addon-agent-cf69d56-bh7fz\" (UID: \"af35c088-4a35-47a4-a2a8-d97d9826c925\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"
Jul 28 17:11:42.115765 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.115505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xfgq\" (UniqueName: \"kubernetes.io/projected/ed04b056-8c18-4a8e-8c9e-53bb72ed6c6a-kube-api-access-5xfgq\") pod \"downloads-7c97dc4555-mr525\" (UID: \"ed04b056-8c18-4a8e-8c9e-53bb72ed6c6a\") " pod="openshift-console/downloads-7c97dc4555-mr525"
Jul 28 17:11:42.115765 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.115545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af35c088-4a35-47a4-a2a8-d97d9826c925-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cf69d56-bh7fz\" (UID: \"af35c088-4a35-47a4-a2a8-d97d9826c925\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"
Jul 28 17:11:42.118652 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.118611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af35c088-4a35-47a4-a2a8-d97d9826c925-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cf69d56-bh7fz\" (UID: \"af35c088-4a35-47a4-a2a8-d97d9826c925\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"
Jul 28 17:11:42.152513 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.152417    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-776cdb854c-qdpln"]
Jul 28 17:11:42.157615 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.157543    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.179435 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.179366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w589v\" (UniqueName: \"kubernetes.io/projected/af35c088-4a35-47a4-a2a8-d97d9826c925-kube-api-access-w589v\") pod \"managed-serviceaccount-addon-agent-cf69d56-bh7fz\" (UID: \"af35c088-4a35-47a4-a2a8-d97d9826c925\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"
Jul 28 17:11:42.215843 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.215817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-trusted-ca\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.215957 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.215879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfgq\" (UniqueName: \"kubernetes.io/projected/ed04b056-8c18-4a8e-8c9e-53bb72ed6c6a-kube-api-access-5xfgq\") pod \"downloads-7c97dc4555-mr525\" (UID: \"ed04b056-8c18-4a8e-8c9e-53bb72ed6c6a\") " pod="openshift-console/downloads-7c97dc4555-mr525"
Jul 28 17:11:42.215957 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.215909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-image-registry-private-configuration\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.215957 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.215941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-installation-pull-secrets\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.216118 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.215969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-registry-tls\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.216118 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.216060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-bound-sa-token\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.216118 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.216108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-registry-certificates\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.216292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.216194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66r5j\" (UniqueName: \"kubernetes.io/projected/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-kube-api-access-66r5j\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.216292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.216227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-ca-trust-extracted\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.217751 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.217695    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-776cdb854c-qdpln"]
Jul 28 17:11:42.224328 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.224102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.244608 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.244588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"
Jul 28 17:11:42.255114 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.255092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfgq\" (UniqueName: \"kubernetes.io/projected/ed04b056-8c18-4a8e-8c9e-53bb72ed6c6a-kube-api-access-5xfgq\") pod \"downloads-7c97dc4555-mr525\" (UID: \"ed04b056-8c18-4a8e-8c9e-53bb72ed6c6a\") " pod="openshift-console/downloads-7c97dc4555-mr525"
Jul 28 17:11:42.317626 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.317522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-trusted-ca\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.317626 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.317561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-image-registry-private-configuration\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.317626 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.317592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-installation-pull-secrets\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.317626 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.317618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-registry-tls\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.317886 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.317706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-bound-sa-token\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.317886 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.317743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-registry-certificates\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.317886 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.317807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66r5j\" (UniqueName: \"kubernetes.io/projected/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-kube-api-access-66r5j\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.317886 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.317834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-ca-trust-extracted\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.319744 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.319065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-ca-trust-extracted\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.320035 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.319951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-trusted-ca\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.320035 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.320000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-registry-certificates\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.320747 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.320709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-installation-pull-secrets\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.321074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.321044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-image-registry-private-configuration\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.321684 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.321664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-registry-tls\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.342852 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.342733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-mr525"
Jul 28 17:11:42.347339 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.347198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66r5j\" (UniqueName: \"kubernetes.io/projected/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-kube-api-access-66r5j\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.352698 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.352651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52f18dc8-7cba-445b-b8d8-a1b68bfe7d37-bound-sa-token\") pod \"image-registry-776cdb854c-qdpln\" (UID: \"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37\") " pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.404963 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.404725    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz"]
Jul 28 17:11:42.405776 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.405736    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.412872 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:42.412840    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf35c088_4a35_47a4_a2a8_d97d9826c925.slice/crio-0149b84e2384eb436cb8aa3f1b8d4216a03df568bacc97118f9e9f36d7fe50a9 WatchSource:0}: Error finding container 0149b84e2384eb436cb8aa3f1b8d4216a03df568bacc97118f9e9f36d7fe50a9: Status 404 returned error can't find the container with id 0149b84e2384eb436cb8aa3f1b8d4216a03df568bacc97118f9e9f36d7fe50a9
Jul 28 17:11:42.471257 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.471228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:42.496115 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.496089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.503671 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.503598    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-mr525"]
Jul 28 17:11:42.505371 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:42.505324    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded04b056_8c18_4a8e_8c9e_53bb72ed6c6a.slice/crio-d5e9687286043748f633b9d68981796824145769846012ffa8cfb658d4efdccb WatchSource:0}: Error finding container d5e9687286043748f633b9d68981796824145769846012ffa8cfb658d4efdccb: Status 404 returned error can't find the container with id d5e9687286043748f633b9d68981796824145769846012ffa8cfb658d4efdccb
Jul 28 17:11:42.549124 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.549097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-mr525" event={"ID":"ed04b056-8c18-4a8e-8c9e-53bb72ed6c6a","Type":"ContainerStarted","Data":"d5e9687286043748f633b9d68981796824145769846012ffa8cfb658d4efdccb"}
Jul 28 17:11:42.551119 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.551080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz" event={"ID":"af35c088-4a35-47a4-a2a8-d97d9826c925","Type":"ContainerStarted","Data":"0149b84e2384eb436cb8aa3f1b8d4216a03df568bacc97118f9e9f36d7fe50a9"}
Jul 28 17:11:42.554692 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.554395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zvdqk" event={"ID":"20e8d68a-03b6-46f5-8500-1fd50954ac27","Type":"ContainerStarted","Data":"01b12be9dbadf1af9d4de3e08da6823f291c230976d9d671ffd49ef764395970"}
Jul 28 17:11:42.554692 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.554494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zvdqk" event={"ID":"20e8d68a-03b6-46f5-8500-1fd50954ac27","Type":"ContainerStarted","Data":"bc4527bf4c9dcc4d69c3936369a73388ac834fd8fbb56b1ba3aa441ffca73bd8"}
Jul 28 17:11:42.559006 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.558983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh" event={"ID":"20744d22-0573-4748-8d9c-75d2761cd691","Type":"ContainerStarted","Data":"f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1"}
Jul 28 17:11:42.559106 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.559027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh" event={"ID":"20744d22-0573-4748-8d9c-75d2761cd691","Type":"ContainerStarted","Data":"63ddd9f3d985adc7466a68763ffc3e4588b72caec631cfcbf80b1e6be6c5aef1"}
Jul 28 17:11:42.559428 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.559361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:11:42.586880 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.585968    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zvdqk" podStartSLOduration=67.588188999 podStartE2EDuration="1m8.585949843s" totalImagesPullingTime="997.760844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:42.585295559 +0000 UTC m=+68.820261011" watchObservedRunningTime="2026-07-28 17:11:42.585949843 +0000 UTC m=+68.820915297"
Jul 28 17:11:42.602367 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.602280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.623669 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.623598    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh" podStartSLOduration=34.623582228 podStartE2EDuration="34.623582228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:42.623302527 +0000 UTC m=+68.858268003" watchObservedRunningTime="2026-07-28 17:11:42.623582228 +0000 UTC m=+68.858547685"
Jul 28 17:11:42.646912 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.646886    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"]
Jul 28 17:11:42.651312 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.651285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"
Jul 28 17:11:42.653603 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.653547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-776cdb854c-qdpln"]
Jul 28 17:11:42.654564 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.654542    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 17:11:42.654652 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.654541    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8bxn5\""
Jul 28 17:11:42.658033 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:42.658003    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f18dc8_7cba_445b_b8d8_a1b68bfe7d37.slice/crio-7a3fca1584d725461d9c2cc362b859156fac0ace4f98a748babdcb8900ed6bef WatchSource:0}: Error finding container 7a3fca1584d725461d9c2cc362b859156fac0ace4f98a748babdcb8900ed6bef: Status 404 returned error can't find the container with id 7a3fca1584d725461d9c2cc362b859156fac0ace4f98a748babdcb8900ed6bef
Jul 28 17:11:42.664152 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.664109    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"]
Jul 28 17:11:42.706362 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.706291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.723526 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.723502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6461a055-c2e9-4c93-a2b1-c2434ab41091-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-knbzg\" (UID: \"6461a055-c2e9-4c93-a2b1-c2434ab41091\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"
Jul 28 17:11:42.824580 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.824512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6461a055-c2e9-4c93-a2b1-c2434ab41091-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-knbzg\" (UID: \"6461a055-c2e9-4c93-a2b1-c2434ab41091\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"
Jul 28 17:11:42.827157 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.827133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6461a055-c2e9-4c93-a2b1-c2434ab41091-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-knbzg\" (UID: \"6461a055-c2e9-4c93-a2b1-c2434ab41091\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"
Jul 28 17:11:42.965533 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.965443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"
Jul 28 17:11:42.988663 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:42.988570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:43.387832 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:43.387793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:43.464934 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:43.464900    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:43.564569 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:43.564515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-776cdb854c-qdpln" event={"ID":"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37","Type":"ContainerStarted","Data":"6e8897afa3e5ffe4b5774240cc8e9b604e34a9563af7b3a5b6d86727c5dff957"}
Jul 28 17:11:43.564733 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:43.564611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-776cdb854c-qdpln" event={"ID":"52f18dc8-7cba-445b-b8d8-a1b68bfe7d37","Type":"ContainerStarted","Data":"7a3fca1584d725461d9c2cc362b859156fac0ace4f98a748babdcb8900ed6bef"}
Jul 28 17:11:43.564733 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:43.564721    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:11:43.587610 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:43.587565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-776cdb854c-qdpln" podStartSLOduration=1.5875469899999999 podStartE2EDuration="1.58754699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:43.585810511 +0000 UTC m=+69.820775963" watchObservedRunningTime="2026-07-28 17:11:43.58754699 +0000 UTC m=+69.822512423"
Jul 28 17:11:43.968728 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:43.968689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:44.088114 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:44.087771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:45.576920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:45.576888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t542q" event={"ID":"8124e945-8779-41d3-86f1-726109a4d6d5","Type":"ContainerStarted","Data":"dd0f252a2b83e1c5b73dd4cc406a12075804d9eeaf358f163dcf1d3e179c6447"}
Jul 28 17:11:45.577621 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:45.577600    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:11:45.604380 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:45.603183    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t542q" podStartSLOduration=66.61524978 podStartE2EDuration="1m11.60316361s" totalImagesPullingTime="4.98791383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:45.602078852 +0000 UTC m=+71.837044306" watchObservedRunningTime="2026-07-28 17:11:45.60316361 +0000 UTC m=+71.838129064"
Jul 28 17:11:45.644668 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:45.644535    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"]
Jul 28 17:11:45.649462 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:45.649428    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6461a055_c2e9_4c93_a2b1_c2434ab41091.slice/crio-ec4877fbe9dfa25acde5129af9c8162c1d0277ddfc3bdd08bcb806ebee5a45e5 WatchSource:0}: Error finding container ec4877fbe9dfa25acde5129af9c8162c1d0277ddfc3bdd08bcb806ebee5a45e5: Status 404 returned error can't find the container with id ec4877fbe9dfa25acde5129af9c8162c1d0277ddfc3bdd08bcb806ebee5a45e5
Jul 28 17:11:45.666092 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:45.666064    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:45.759600 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:45.759527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:45.833090 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:45.833005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:46.582419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:46.582368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg" event={"ID":"6461a055-c2e9-4c93-a2b1-c2434ab41091","Type":"ContainerStarted","Data":"ec4877fbe9dfa25acde5129af9c8162c1d0277ddfc3bdd08bcb806ebee5a45e5"}
Jul 28 17:11:46.584230 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:46.584198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qf77t" event={"ID":"b1684222-25b4-4a1c-b62f-df4ea459ffe9","Type":"ContainerStarted","Data":"1252df02ee67631ea37d121988ef6b4c18ee0ab39619ec64efa13283889e1b3e"}
Jul 28 17:11:46.601354 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:46.601182    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qf77t" podStartSLOduration=67.592035217 podStartE2EDuration="1m12.601163556s" totalImagesPullingTime="5.009128339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:46.600579393 +0000 UTC m=+72.835544848" watchObservedRunningTime="2026-07-28 17:11:46.601163556 +0000 UTC m=+72.836129007"
Jul 28 17:11:47.467645 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:47.467614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:47.469965 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:47.469939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21b940d6-a972-4a7e-b3e2-3054c9bac62f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ffmn\" (UID: \"21b940d6-a972-4a7e-b3e2-3054c9bac62f\") " pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:47.589560 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:47.589526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg" event={"ID":"6461a055-c2e9-4c93-a2b1-c2434ab41091","Type":"ContainerStarted","Data":"81776d04b16fde1e1d48b9e47ac41e31cca9d5e243eda463f9eef72eded5d811"}
Jul 28 17:11:47.589943 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:47.589726    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"
Jul 28 17:11:47.595340 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:47.595317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg"
Jul 28 17:11:47.614864 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:47.614826    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-knbzg" podStartSLOduration=4.483386041 podStartE2EDuration="5.614813109s" totalImagesPullingTime="1.131427068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:47.614709408 +0000 UTC m=+73.849674861" watchObservedRunningTime="2026-07-28 17:11:47.614813109 +0000 UTC m=+73.849778538"
Jul 28 17:11:47.696222 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:47.696194    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rd4np\""
Jul 28 17:11:47.703546 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:47.703522    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6ffmn"
Jul 28 17:11:47.855187 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:47.855157    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6ffmn"]
Jul 28 17:11:47.858213 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:11:47.858186    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b940d6_a972_4a7e_b3e2_3054c9bac62f.slice/crio-758e501f56e52853444701a5aa87b60e103bd5c2d0ffb7fd311d3ab881b38a53 WatchSource:0}: Error finding container 758e501f56e52853444701a5aa87b60e103bd5c2d0ffb7fd311d3ab881b38a53: Status 404 returned error can't find the container with id 758e501f56e52853444701a5aa87b60e103bd5c2d0ffb7fd311d3ab881b38a53
Jul 28 17:11:48.598502 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:48.598466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ffmn" event={"ID":"21b940d6-a972-4a7e-b3e2-3054c9bac62f","Type":"ContainerStarted","Data":"758e501f56e52853444701a5aa87b60e103bd5c2d0ffb7fd311d3ab881b38a53"}
Jul 28 17:11:48.671930 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:48.671897    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:48.773943 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:48.773865    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:49.094483 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:49.094339    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:49.159842 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:49.159809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.038340 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:51.038303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.101141 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:51.101109    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.383307 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:51.383265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.483173 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:51.483138    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:53.688516 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:53.688431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:53.799538 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:53.799430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:53.915056 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:53.915014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:54.414417 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:54.414376    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:54.624448 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:54.624391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz" event={"ID":"af35c088-4a35-47a4-a2a8-d97d9826c925","Type":"ContainerStarted","Data":"8e538c328e74634de1adae7a0884fb3e5c920bc997c55afe6b7ece9394d0b52c"}
Jul 28 17:11:54.626078 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:54.626048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ffmn" event={"ID":"21b940d6-a972-4a7e-b3e2-3054c9bac62f","Type":"ContainerStarted","Data":"d3ea4a45b3444ac7cff2b10407eb82ad7ed172a2d9a7071b366c82445fae9b8f"}
Jul 28 17:11:54.646865 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:54.646804    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cf69d56-bh7fz" podStartSLOduration=2.133834087 podStartE2EDuration="13.646785287s" totalImagesPullingTime="11.5129512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:54.645566756 +0000 UTC m=+80.880532207" watchObservedRunningTime="2026-07-28 17:11:54.646785287 +0000 UTC m=+80.881750740"
Jul 28 17:11:54.892669 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:54.892624    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:55.835292 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:55.835253    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.058726 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.058644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.080657 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.080542    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"]
Jul 28 17:11:56.084126 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.084108    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"]
Jul 28 17:11:56.084243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.084224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.088263 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.087065    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 17:11:56.088263 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.087354    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:11:56.088263 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.087656    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:11:56.088263 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.087841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.089150 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.089130    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4r7n7\""
Jul 28 17:11:56.089365 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.089347    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:11:56.091202 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.091182    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9q5sz"]
Jul 28 17:11:56.092212 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.091665    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 17:11:56.092212 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.091761    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:11:56.092212 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.091877    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 17:11:56.092212 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.091961    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7vtrq\""
Jul 28 17:11:56.092212 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.092078    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:11:56.094966 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.094920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.100384 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.100315    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"]
Jul 28 17:11:56.104889 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.104870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"]
Jul 28 17:11:56.105305 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.105284    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:11:56.105796 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.105509    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pzgw5\""
Jul 28 17:11:56.105796 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.105619    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:11:56.105796 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.105699    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:11:56.165362 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.165313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-textfile\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzm4r\" (UniqueName: \"kubernetes.io/projected/f927c5f4-d7f2-47a3-8738-5520c084b7d7-kube-api-access-tzm4r\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-tls\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc8b876-28fc-4f87-beb3-694d36ee6c14-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-wtmp\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f927c5f4-d7f2-47a3-8738-5520c084b7d7-sys\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.238648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f927c5f4-d7f2-47a3-8738-5520c084b7d7-root\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.239269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrkq\" (UniqueName: \"kubernetes.io/projected/9cc8b876-28fc-4f87-beb3-694d36ee6c14-kube-api-access-2wrkq\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.239269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4b77928a-0f53-4b14-9cc0-02761f3299cb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.239269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b77928a-0f53-4b14-9cc0-02761f3299cb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.239269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.239269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cc8b876-28fc-4f87-beb3-694d36ee6c14-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.239269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f927c5f4-d7f2-47a3-8738-5520c084b7d7-metrics-client-ca\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.239269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phflj\" (UniqueName: \"kubernetes.io/projected/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-api-access-phflj\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.239269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.238883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cc8b876-28fc-4f87-beb3-694d36ee6c14-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.258200 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.258170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.339613 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.339533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f927c5f4-d7f2-47a3-8738-5520c084b7d7-metrics-client-ca\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.339613 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.339574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phflj\" (UniqueName: \"kubernetes.io/projected/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-api-access-phflj\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.339613 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.339607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cc8b876-28fc-4f87-beb3-694d36ee6c14-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.339874 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.339638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-textfile\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.339992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzm4r\" (UniqueName: \"kubernetes.io/projected/f927c5f4-d7f2-47a3-8738-5520c084b7d7-kube-api-access-tzm4r\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-tls\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc8b876-28fc-4f87-beb3-694d36ee6c14-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-wtmp\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f927c5f4-d7f2-47a3-8738-5520c084b7d7-sys\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f927c5f4-d7f2-47a3-8738-5520c084b7d7-metrics-client-ca\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:56.340314    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:56.340376    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-tls podName:f927c5f4-d7f2-47a3-8738-5520c084b7d7 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:56.840357603 +0000 UTC m=+83.075323039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-tls") pod "node-exporter-9q5sz" (UID: "f927c5f4-d7f2-47a3-8738-5520c084b7d7") : secret "node-exporter-tls" not found
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f927c5f4-d7f2-47a3-8738-5520c084b7d7-sys\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc8b876-28fc-4f87-beb3-694d36ee6c14-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.340987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.341001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-wtmp\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.341079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f927c5f4-d7f2-47a3-8738-5520c084b7d7-root\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.341630 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:56.341093    2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 17:11:56.342597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.341124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrkq\" (UniqueName: \"kubernetes.io/projected/9cc8b876-28fc-4f87-beb3-694d36ee6c14-kube-api-access-2wrkq\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.342597 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:56.341152    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-tls podName:4b77928a-0f53-4b14-9cc0-02761f3299cb nodeName:}" failed. No retries permitted until 2026-07-28 17:11:56.841135212 +0000 UTC m=+83.076100649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-klk6b" (UID: "4b77928a-0f53-4b14-9cc0-02761f3299cb") : secret "kube-state-metrics-tls" not found
Jul 28 17:11:56.342597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.341167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f927c5f4-d7f2-47a3-8738-5520c084b7d7-root\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.342597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.341172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4b77928a-0f53-4b14-9cc0-02761f3299cb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.342597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.341205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b77928a-0f53-4b14-9cc0-02761f3299cb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.342597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.341246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.342597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.341277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cc8b876-28fc-4f87-beb3-694d36ee6c14-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.342597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.341520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4b77928a-0f53-4b14-9cc0-02761f3299cb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.342597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.342049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b77928a-0f53-4b14-9cc0-02761f3299cb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.343042 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.342626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.343445 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.343153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.343445 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.343331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-textfile\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.344333 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.344232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.344333 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.344267    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cc8b876-28fc-4f87-beb3-694d36ee6c14-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.352434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.351975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.352434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.352109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cc8b876-28fc-4f87-beb3-694d36ee6c14-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.355996 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.355908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrkq\" (UniqueName: \"kubernetes.io/projected/9cc8b876-28fc-4f87-beb3-694d36ee6c14-kube-api-access-2wrkq\") pod \"openshift-state-metrics-b8f49f5c6-mzwvz\" (UID: \"9cc8b876-28fc-4f87-beb3-694d36ee6c14\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.356236 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.356215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phflj\" (UniqueName: \"kubernetes.io/projected/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-api-access-phflj\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.359783 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.359174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzm4r\" (UniqueName: \"kubernetes.io/projected/f927c5f4-d7f2-47a3-8738-5520c084b7d7-kube-api-access-tzm4r\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.361275 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.361116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.417611 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.417393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"
Jul 28 17:11:56.459901 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.459874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.559978 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.559921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.702717 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.702613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.760221 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.760187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.846302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.846268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:56.846481 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.846359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-tls\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.846481 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:56.846392    2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 17:11:56.846606 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:11:56.846482    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-tls podName:4b77928a-0f53-4b14-9cc0-02761f3299cb nodeName:}" failed. No retries permitted until 2026-07-28 17:11:57.846458272 +0000 UTC m=+84.081423704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-klk6b" (UID: "4b77928a-0f53-4b14-9cc0-02761f3299cb") : secret "kube-state-metrics-tls" not found
Jul 28 17:11:56.848956 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.848932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f927c5f4-d7f2-47a3-8738-5520c084b7d7-node-exporter-tls\") pod \"node-exporter-9q5sz\" (UID: \"f927c5f4-d7f2-47a3-8738-5520c084b7d7\") " pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:56.859723 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.859653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.974826 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:56.974753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:57.026989 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:57.026956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9q5sz"
Jul 28 17:11:57.099526 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:57.099489    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:57.191862 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:57.191821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:57.307054 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:57.307016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:57.431993 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:57.431954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:57.854774 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:57.854732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:57.857655 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:57.857621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b77928a-0f53-4b14-9cc0-02761f3299cb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-klk6b\" (UID: \"4b77928a-0f53-4b14-9cc0-02761f3299cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:57.907520 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:57.907494    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"
Jul 28 17:11:58.145517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:58.145424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:58.668153 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:58.668115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:59.044935 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:59.044825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:59.106973 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:59.106942    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:59.724598 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:59.724556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:59.828011 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:11:59.827964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:00.180382 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.180345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:00.277533 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.277498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:00.393871 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.393839    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6788c89f75-58rcg"]
Jul 28 17:12:00.445073 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.445004    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6788c89f75-58rcg"]
Jul 28 17:12:00.445215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.445090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.449169 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.449136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wt6qc\""
Jul 28 17:12:00.449169 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.449152    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:12:00.449460 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.449442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 17:12:00.449535 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.449511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4pgfjlk7nvmia\""
Jul 28 17:12:00.449668 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.449451    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 17:12:00.449795 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.449655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 17:12:00.575791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.575750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-audit-log\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.575971 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.575806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-secret-metrics-server-client-certs\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.575971 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.575841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-metrics-server-audit-profiles\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.575971 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.575865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-client-ca-bundle\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.575971 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.575907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-secret-metrics-server-tls\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.575971 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.575964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.576175 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.575998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5w9w\" (UniqueName: \"kubernetes.io/projected/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-kube-api-access-g5w9w\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.677194 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.677161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-audit-log\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.677363 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.677217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-secret-metrics-server-client-certs\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.677363 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.677254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-metrics-server-audit-profiles\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.677363 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.677278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-client-ca-bundle\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.677363 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.677318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-secret-metrics-server-tls\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.677530 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.677379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.677530 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.677414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w9w\" (UniqueName: \"kubernetes.io/projected/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-kube-api-access-g5w9w\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.677594 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.677559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-audit-log\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.678517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.678318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.678757 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.678717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-metrics-server-audit-profiles\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.680265 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.680240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-secret-metrics-server-client-certs\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.680382 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.680296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-client-ca-bundle\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.680452 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.680386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-secret-metrics-server-tls\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.686536 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.686514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5w9w\" (UniqueName: \"kubernetes.io/projected/993984eb-49eb-4fc4-9c97-fdfcf3d07a7e-kube-api-access-g5w9w\") pod \"metrics-server-6788c89f75-58rcg\" (UID: \"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e\") " pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.756089 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.756022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:00.848497 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.848465    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:00.912046 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:00.912014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:01.047385 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.047327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:01.154323 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.154281    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:01.301227 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.301146    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-f884997dd-7crk9"]
Jul 28 17:12:01.313935 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.313912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.316743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.316591    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 17:12:01.316743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.316615    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 17:12:01.316743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.316633    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 17:12:01.316743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.316643    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 17:12:01.317034 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.317016    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 17:12:01.317423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.317389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-777vt\""
Jul 28 17:12:01.321421 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.321243    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f884997dd-7crk9"]
Jul 28 17:12:01.324184 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.324165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 17:12:01.393239 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.393206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:01.484967 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.484930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-serving-certs-ca-bundle\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.484967 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.484972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-metrics-client-ca\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.485299 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.485022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-secret-telemeter-client\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.485299 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.485052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.485299 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.485086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-federate-client-tls\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.485299 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.485108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4jk\" (UniqueName: \"kubernetes.io/projected/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-kube-api-access-ff4jk\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.485299 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.485134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-telemeter-client-tls\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.485299 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.485160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.585959 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.585858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-serving-certs-ca-bundle\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.585959 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.585908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-metrics-client-ca\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.585959 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.585950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-secret-telemeter-client\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.586243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.585985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.586243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.586018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-federate-client-tls\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.586243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.586040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4jk\" (UniqueName: \"kubernetes.io/projected/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-kube-api-access-ff4jk\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.586243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.586076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-telemeter-client-tls\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.586243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.586138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.587040 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.586785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-serving-certs-ca-bundle\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.587040 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.586804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-metrics-client-ca\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.587394 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.587366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.589130 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.589097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-secret-telemeter-client\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.589233 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.589156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-federate-client-tls\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.589349 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.589322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.589349 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.589341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-telemeter-client-tls\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.596453 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.596431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4jk\" (UniqueName: \"kubernetes.io/projected/3af9ca0c-a879-42ff-aca1-e5c7fc8bac90-kube-api-access-ff4jk\") pod \"telemeter-client-f884997dd-7crk9\" (UID: \"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90\") " pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.628187 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.628155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f884997dd-7crk9"
Jul 28 17:12:01.875912 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:01.875822    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f884997dd-7crk9"]
Jul 28 17:12:01.879662 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:12:01.879634    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af9ca0c_a879_42ff_aca1_e5c7fc8bac90.slice/crio-a7bed1270fe62d83904beca69a65121f235c27dce6ec2b0358a33bbe6765d51a WatchSource:0}: Error finding container a7bed1270fe62d83904beca69a65121f235c27dce6ec2b0358a33bbe6765d51a: Status 404 returned error can't find the container with id a7bed1270fe62d83904beca69a65121f235c27dce6ec2b0358a33bbe6765d51a
Jul 28 17:12:02.100627 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.100589    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6788c89f75-58rcg"]
Jul 28 17:12:02.103766 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.103698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b"]
Jul 28 17:12:02.105912 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.105684    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz"]
Jul 28 17:12:02.105912 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:12:02.105733    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993984eb_49eb_4fc4_9c97_fdfcf3d07a7e.slice/crio-a506ac4a8098a710cc0578771ffba6abd1f523c307fe3889e21dd197b92cc3b6 WatchSource:0}: Error finding container a506ac4a8098a710cc0578771ffba6abd1f523c307fe3889e21dd197b92cc3b6: Status 404 returned error can't find the container with id a506ac4a8098a710cc0578771ffba6abd1f523c307fe3889e21dd197b92cc3b6
Jul 28 17:12:02.106500 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:12:02.106334    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc8b876_28fc_4f87_beb3_694d36ee6c14.slice/crio-edb45dc63c8b169fb766aaa0c26947b92fc1c56452c6f6ac6d5e51ded2f626bc WatchSource:0}: Error finding container edb45dc63c8b169fb766aaa0c26947b92fc1c56452c6f6ac6d5e51ded2f626bc: Status 404 returned error can't find the container with id edb45dc63c8b169fb766aaa0c26947b92fc1c56452c6f6ac6d5e51ded2f626bc
Jul 28 17:12:02.107881 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:12:02.107855    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b77928a_0f53_4b14_9cc0_02761f3299cb.slice/crio-cca6c05c889897748e601063eb81c7a388b34fbeff22371ef7859fe94f44240c WatchSource:0}: Error finding container cca6c05c889897748e601063eb81c7a388b34fbeff22371ef7859fe94f44240c: Status 404 returned error can't find the container with id cca6c05c889897748e601063eb81c7a388b34fbeff22371ef7859fe94f44240c
Jul 28 17:12:02.356194 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.356115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:02.641649 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.641592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:02.665338 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.665256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b" event={"ID":"4b77928a-0f53-4b14-9cc0-02761f3299cb","Type":"ContainerStarted","Data":"cca6c05c889897748e601063eb81c7a388b34fbeff22371ef7859fe94f44240c"}
Jul 28 17:12:02.668757 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.668678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz" event={"ID":"9cc8b876-28fc-4f87-beb3-694d36ee6c14","Type":"ContainerStarted","Data":"4d9fb75538942d8e929df4365f0467901ef3e311f2ac2417ff55a554c70b9829"}
Jul 28 17:12:02.668757 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.668737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz" event={"ID":"9cc8b876-28fc-4f87-beb3-694d36ee6c14","Type":"ContainerStarted","Data":"19b9bf0001516cc9b3ec4f5989d01a5a9b6505cd0247fc37177eef73e238485b"}
Jul 28 17:12:02.668953 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.668762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz" event={"ID":"9cc8b876-28fc-4f87-beb3-694d36ee6c14","Type":"ContainerStarted","Data":"edb45dc63c8b169fb766aaa0c26947b92fc1c56452c6f6ac6d5e51ded2f626bc"}
Jul 28 17:12:02.671118 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.670822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6788c89f75-58rcg" event={"ID":"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e","Type":"ContainerStarted","Data":"a506ac4a8098a710cc0578771ffba6abd1f523c307fe3889e21dd197b92cc3b6"}
Jul 28 17:12:02.676112 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.676046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q5sz" event={"ID":"f927c5f4-d7f2-47a3-8738-5520c084b7d7","Type":"ContainerStarted","Data":"c07fed37fc5f4ebc4b772a1b020ef0a02978bed7724c02c4d10c824be122d17c"}
Jul 28 17:12:02.692924 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.690300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-mr525" event={"ID":"ed04b056-8c18-4a8e-8c9e-53bb72ed6c6a","Type":"ContainerStarted","Data":"49e99793f3f79a41f6a1874f96a3c713e04e3370b2c4d8ccc768d737339e9ee4"}
Jul 28 17:12:02.692924 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.691326    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-mr525"
Jul 28 17:12:02.696580 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.696004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f884997dd-7crk9" event={"ID":"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90","Type":"ContainerStarted","Data":"a7bed1270fe62d83904beca69a65121f235c27dce6ec2b0358a33bbe6765d51a"}
Jul 28 17:12:02.704338 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.704252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:02.709291 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.709225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-mr525"
Jul 28 17:12:02.713905 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:02.712977    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-mr525" podStartSLOduration=2.194292581 podStartE2EDuration="21.712961454s" totalImagesPullingTime="19.518668873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:02.711236926 +0000 UTC m=+88.946202380" watchObservedRunningTime="2026-07-28 17:12:02.712961454 +0000 UTC m=+88.947926907"
Jul 28 17:12:03.575118 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:03.574885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:12:03.578201 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:03.578132    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:03.732651 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:03.732442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:04.573713 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:04.573683    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-776cdb854c-qdpln"
Jul 28 17:12:04.707085 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:04.706949    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:04.779904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:04.779744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:08.585355 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:08.585302    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh" podUID="20744d22-0573-4748-8d9c-75d2761cd691" containerName="registry" containerID="cri-o://f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1" gracePeriod=30
Jul 28 17:12:09.700172 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.696702    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:12:09.730418 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.730371    2579 generic.go:356] "Generic (PLEG): container finished" podID="20744d22-0573-4748-8d9c-75d2761cd691" containerID="f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1" exitCode=0
Jul 28 17:12:09.730557 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.730536    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh"
Jul 28 17:12:09.730715 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.730694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh" event={"ID":"20744d22-0573-4748-8d9c-75d2761cd691","Type":"ContainerDied","Data":"f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1"}
Jul 28 17:12:09.730888 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.730871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67d785cb4c-dhvrh" event={"ID":"20744d22-0573-4748-8d9c-75d2761cd691","Type":"ContainerDied","Data":"63ddd9f3d985adc7466a68763ffc3e4588b72caec631cfcbf80b1e6be6c5aef1"}
Jul 28 17:12:09.731005 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.730989    2579 scope.go:117] "RemoveContainer" containerID="f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1"
Jul 28 17:12:09.744655 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.744622    2579 scope.go:117] "RemoveContainer" containerID="f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1"
Jul 28 17:12:09.745062 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:12:09.745034    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1\": container with ID starting with f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1 not found: ID does not exist" containerID="f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1"
Jul 28 17:12:09.745158 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.745073    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1"} err="failed to get container status \"f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1\": rpc error: code = NotFound desc = could not find container \"f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1\": container with ID starting with f9ed6dca589c3c06760f46fa91d7ca245d3a34c3cf7e0f202cca755b407c4ba1 not found: ID does not exist"
Jul 28 17:12:09.765728 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.765704    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-bound-sa-token\") pod \"20744d22-0573-4748-8d9c-75d2761cd691\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") "
Jul 28 17:12:09.765912 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.765894    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-installation-pull-secrets\") pod \"20744d22-0573-4748-8d9c-75d2761cd691\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") "
Jul 28 17:12:09.766057 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.766042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20744d22-0573-4748-8d9c-75d2761cd691-ca-trust-extracted\") pod \"20744d22-0573-4748-8d9c-75d2761cd691\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") "
Jul 28 17:12:09.766422 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.766378    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") pod \"20744d22-0573-4748-8d9c-75d2761cd691\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") "
Jul 28 17:12:09.766574 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.766557    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-registry-certificates\") pod \"20744d22-0573-4748-8d9c-75d2761cd691\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") "
Jul 28 17:12:09.766692 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.766676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-trusted-ca\") pod \"20744d22-0573-4748-8d9c-75d2761cd691\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") "
Jul 28 17:12:09.767286 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.766794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqcqp\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-kube-api-access-wqcqp\") pod \"20744d22-0573-4748-8d9c-75d2761cd691\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") "
Jul 28 17:12:09.767286 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.766835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-image-registry-private-configuration\") pod \"20744d22-0573-4748-8d9c-75d2761cd691\" (UID: \"20744d22-0573-4748-8d9c-75d2761cd691\") "
Jul 28 17:12:09.768252 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.768199    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "20744d22-0573-4748-8d9c-75d2761cd691" (UID: "20744d22-0573-4748-8d9c-75d2761cd691"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:09.768636 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.768592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "20744d22-0573-4748-8d9c-75d2761cd691" (UID: "20744d22-0573-4748-8d9c-75d2761cd691"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:09.774350 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.774295    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "20744d22-0573-4748-8d9c-75d2761cd691" (UID: "20744d22-0573-4748-8d9c-75d2761cd691"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:12:09.774717 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.774619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "20744d22-0573-4748-8d9c-75d2761cd691" (UID: "20744d22-0573-4748-8d9c-75d2761cd691"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:12:09.775254 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.775219    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "20744d22-0573-4748-8d9c-75d2761cd691" (UID: "20744d22-0573-4748-8d9c-75d2761cd691"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:12:09.775499 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.775468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "20744d22-0573-4748-8d9c-75d2761cd691" (UID: "20744d22-0573-4748-8d9c-75d2761cd691"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:12:09.776314 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.776228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-kube-api-access-wqcqp" (OuterVolumeSpecName: "kube-api-access-wqcqp") pod "20744d22-0573-4748-8d9c-75d2761cd691" (UID: "20744d22-0573-4748-8d9c-75d2761cd691"). InnerVolumeSpecName "kube-api-access-wqcqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:12:09.781615 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.781593    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20744d22-0573-4748-8d9c-75d2761cd691-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "20744d22-0573-4748-8d9c-75d2761cd691" (UID: "20744d22-0573-4748-8d9c-75d2761cd691"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:12:09.869074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.868931    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-installation-pull-secrets\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:12:09.869074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.868964    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20744d22-0573-4748-8d9c-75d2761cd691-ca-trust-extracted\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:12:09.869074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.868980    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-registry-tls\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:12:09.869074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.869001    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-registry-certificates\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:12:09.869074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.869016    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20744d22-0573-4748-8d9c-75d2761cd691-trusted-ca\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:12:09.869074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.869028    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqcqp\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-kube-api-access-wqcqp\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:12:09.869074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.869041    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20744d22-0573-4748-8d9c-75d2761cd691-image-registry-private-configuration\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:12:09.869074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:09.869054    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20744d22-0573-4748-8d9c-75d2761cd691-bound-sa-token\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:12:10.056365 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.056313    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67d785cb4c-dhvrh"]
Jul 28 17:12:10.061649 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.061619    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67d785cb4c-dhvrh"]
Jul 28 17:12:10.285999 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.285925    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20744d22-0573-4748-8d9c-75d2761cd691" path="/var/lib/kubelet/pods/20744d22-0573-4748-8d9c-75d2761cd691/volumes"
Jul 28 17:12:10.740660 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.740611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ffmn" event={"ID":"21b940d6-a972-4a7e-b3e2-3054c9bac62f","Type":"ContainerStarted","Data":"3237c03acd34b406e3aaf17b01712fd88621f237839c6de806859e7098b1f030"}
Jul 28 17:12:10.746692 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.746618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b" event={"ID":"4b77928a-0f53-4b14-9cc0-02761f3299cb","Type":"ContainerStarted","Data":"29ebfba42f39eaa808972f7b58b892bf7686939a0f96d9d64a5d61868bd6701a"}
Jul 28 17:12:10.746692 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.746677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b" event={"ID":"4b77928a-0f53-4b14-9cc0-02761f3299cb","Type":"ContainerStarted","Data":"6310a6e39a108fdc0affea200c964d06021d9f8fbf017683348b9d57c6542977"}
Jul 28 17:12:10.746917 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.746709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b" event={"ID":"4b77928a-0f53-4b14-9cc0-02761f3299cb","Type":"ContainerStarted","Data":"8b62ed0526401aafb58c5fd187384d10a3c541736b196d44921d3efed990258c"}
Jul 28 17:12:10.750266 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.750234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz" event={"ID":"9cc8b876-28fc-4f87-beb3-694d36ee6c14","Type":"ContainerStarted","Data":"4509f51254dba0c7ad4887164bbabe7d6202e27d0422484bf8a81dd4f7e5d216"}
Jul 28 17:12:10.752889 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.752832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6788c89f75-58rcg" event={"ID":"993984eb-49eb-4fc4-9c97-fdfcf3d07a7e","Type":"ContainerStarted","Data":"d304afbdf6d64d28a6ee122c9b20ab3fe988dac1b388de0b61ed973119032aaf"}
Jul 28 17:12:10.755692 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.755663    2579 generic.go:356] "Generic (PLEG): container finished" podID="f927c5f4-d7f2-47a3-8738-5520c084b7d7" containerID="22612c8805a3fa1dc6b4d7053f6c4be050d86b8f03c514a9077cd88cd358b4a9" exitCode=0
Jul 28 17:12:10.755921 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.755890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q5sz" event={"ID":"f927c5f4-d7f2-47a3-8738-5520c084b7d7","Type":"ContainerDied","Data":"22612c8805a3fa1dc6b4d7053f6c4be050d86b8f03c514a9077cd88cd358b4a9"}
Jul 28 17:12:10.758386 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.758353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f884997dd-7crk9" event={"ID":"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90","Type":"ContainerStarted","Data":"3003983b85189b01aa2c7d1d04bf4990cc940c5937f3b88109af03d7d405eb96"}
Jul 28 17:12:10.770436 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.770367    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-klk6b" podStartSLOduration=7.233582707 podStartE2EDuration="14.770350099s" totalImagesPullingTime="7.536767392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:10.768860935 +0000 UTC m=+97.003826378" watchObservedRunningTime="2026-07-28 17:12:10.770350099 +0000 UTC m=+97.005315551"
Jul 28 17:12:10.802272 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.802063    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mzwvz" podStartSLOduration=7.399517491 podStartE2EDuration="14.802043993s" totalImagesPullingTime="7.402526502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:10.800146341 +0000 UTC m=+97.035111796" watchObservedRunningTime="2026-07-28 17:12:10.802043993 +0000 UTC m=+97.037009449"
Jul 28 17:12:10.825608 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:10.825562    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6788c89f75-58rcg" podStartSLOduration=3.285090239 podStartE2EDuration="10.82554868s" totalImagesPullingTime="7.540458441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:10.823309714 +0000 UTC m=+97.058275202" watchObservedRunningTime="2026-07-28 17:12:10.82554868 +0000 UTC m=+97.060514133"
Jul 28 17:12:11.183176 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:11.183134    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:11.287959 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:11.287928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:11.764178 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:11.764147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f884997dd-7crk9" event={"ID":"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90","Type":"ContainerStarted","Data":"a604097d7c206743ed40f3add7d6c0cd72c7e47b21a582d388fe03f8c023dfb2"}
Jul 28 17:12:11.766463 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:11.766430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q5sz" event={"ID":"f927c5f4-d7f2-47a3-8738-5520c084b7d7","Type":"ContainerStarted","Data":"871ad8be1e06915923f9a3aa208c32224869956f76fc4fa1f7c0bc883d1542ad"}
Jul 28 17:12:11.766551 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:11.766478    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q5sz" event={"ID":"f927c5f4-d7f2-47a3-8738-5520c084b7d7","Type":"ContainerStarted","Data":"9fbec5868031afa5a4a7107ba468ed4171965dafe495fbfb81449467c19183c4"}
Jul 28 17:12:11.798415 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:11.798359    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9q5sz" podStartSLOduration=9.475948249 podStartE2EDuration="15.798343446s" totalImagesPullingTime="6.202394393s" totalInitContainerRuntime="120.000804ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:11.796954783 +0000 UTC m=+98.031920236" watchObservedRunningTime="2026-07-28 17:12:11.798343446 +0000 UTC m=+98.033308898"
Jul 28 17:12:12.772582 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:12.772470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f884997dd-7crk9" event={"ID":"3af9ca0c-a879-42ff-aca1-e5c7fc8bac90","Type":"ContainerStarted","Data":"075d930c455e15f51d1bf527c2e3e84e5fefd379711a12dbddf34eca59b47a75"}
Jul 28 17:12:12.816419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:12.816306    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-f884997dd-7crk9" podStartSLOduration=2.5294890949999997 podStartE2EDuration="11.816285463s" totalImagesPullingTime="9.286796368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:12.814205636 +0000 UTC m=+99.049171090" watchObservedRunningTime="2026-07-28 17:12:12.816285463 +0000 UTC m=+99.051250915"
Jul 28 17:12:12.914696 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:12.914661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:13.050445 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:13.050395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:14.066629 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.066599    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c54b6dc66-hm6mv"]
Jul 28 17:12:14.067005 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.066876    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20744d22-0573-4748-8d9c-75d2761cd691" containerName="registry"
Jul 28 17:12:14.067005 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.066887    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="20744d22-0573-4748-8d9c-75d2761cd691" containerName="registry"
Jul 28 17:12:14.067005 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.066929    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="20744d22-0573-4748-8d9c-75d2761cd691" containerName="registry"
Jul 28 17:12:14.083037 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.083012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.087877 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.087853    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:12:14.088222 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.088204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:12:14.091907 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.091890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:12:14.094669 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.094652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:12:14.094726 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.094655    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:12:14.100073 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.100027    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:12:14.104377 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.104357    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l68jz\""
Jul 28 17:12:14.115684 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.115663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-console-config\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.115896 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.115719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-trusted-ca-bundle\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.115896 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.115752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjd9b\" (UniqueName: \"kubernetes.io/projected/e9805559-781e-4157-826b-0a2b43d72c2e-kube-api-access-qjd9b\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.115896 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.115786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-service-ca\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.115896 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.115856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-oauth-serving-cert\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.116072 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.115907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-serving-cert\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.116072 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.115956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-oauth-config\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.151623 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.151598    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c54b6dc66-hm6mv"]
Jul 28 17:12:14.216852 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.216829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-serving-cert\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.216974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.216874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-oauth-config\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.216974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.216906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-console-config\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.216974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.216951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-trusted-ca-bundle\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.217142 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.216992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjd9b\" (UniqueName: \"kubernetes.io/projected/e9805559-781e-4157-826b-0a2b43d72c2e-kube-api-access-qjd9b\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.217142 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.217031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-service-ca\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.217142 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.217055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-oauth-serving-cert\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.217612 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.217588    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-console-config\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.217793 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.217771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-oauth-serving-cert\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.217793 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.217785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-service-ca\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.217942 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.217874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-trusted-ca-bundle\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.219304 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.219283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-serving-cert\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.219416 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.219386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-oauth-config\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.228878 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.228859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjd9b\" (UniqueName: \"kubernetes.io/projected/e9805559-781e-4157-826b-0a2b43d72c2e-kube-api-access-qjd9b\") pod \"console-c54b6dc66-hm6mv\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") " pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.392332 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.392263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:14.554604 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.554579    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c54b6dc66-hm6mv"]
Jul 28 17:12:14.557456 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:12:14.557431    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9805559_781e_4157_826b_0a2b43d72c2e.slice/crio-dc2e770b9f9edd6efb2064bb777c3fc666ae3c916f9b2f0a1f3eefea950527ed WatchSource:0}: Error finding container dc2e770b9f9edd6efb2064bb777c3fc666ae3c916f9b2f0a1f3eefea950527ed: Status 404 returned error can't find the container with id dc2e770b9f9edd6efb2064bb777c3fc666ae3c916f9b2f0a1f3eefea950527ed
Jul 28 17:12:14.668551 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.668470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:14.764302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.764255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:14.781237 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.781207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c54b6dc66-hm6mv" event={"ID":"e9805559-781e-4157-826b-0a2b43d72c2e","Type":"ContainerStarted","Data":"dc2e770b9f9edd6efb2064bb777c3fc666ae3c916f9b2f0a1f3eefea950527ed"}
Jul 28 17:12:14.783236 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.783207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ffmn" event={"ID":"21b940d6-a972-4a7e-b3e2-3054c9bac62f","Type":"ContainerStarted","Data":"d29fc5b8ed3bd9559bf3ad74f316343f08d24bea8efa5b165ee78e99837b7ce2"}
Jul 28 17:12:14.813228 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.813170    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6ffmn" podStartSLOduration=33.4480758 podStartE2EDuration="59.813152405s" totalImagesPullingTime="26.365076605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:14.809791427 +0000 UTC m=+101.044756879" watchObservedRunningTime="2026-07-28 17:12:14.813152405 +0000 UTC m=+101.048117858"
Jul 28 17:12:14.885862 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:14.885832    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:15.064772 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:15.064732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:15.201242 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:15.201205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:17.592322 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:17.592292    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t542q"
Jul 28 17:12:18.559257 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:18.559217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:18.720505 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:18.720471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:18.799947 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:18.799915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c54b6dc66-hm6mv" event={"ID":"e9805559-781e-4157-826b-0a2b43d72c2e","Type":"ContainerStarted","Data":"8e73b5d44dd698a15c4a3be710072c5e58db02548aae3e00719639fb4504f07c"}
Jul 28 17:12:18.820091 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:18.819989    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c54b6dc66-hm6mv" podStartSLOduration=1.467987501 podStartE2EDuration="4.819970953s" totalImagesPullingTime="3.351983452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:18.818948447 +0000 UTC m=+105.053913901" watchObservedRunningTime="2026-07-28 17:12:18.819970953 +0000 UTC m=+105.054936406"
Jul 28 17:12:20.757033 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:20.756996    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:20.757540 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:20.757063    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:24.392410 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:24.392377    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:24.392996 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:24.392445    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:24.397451 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:24.397426    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:24.829524 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:24.829495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:12:25.110246 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:25.110157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:25.203492 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:25.203457    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:25.335514 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:25.335468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:35.542203 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:35.542160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:40.762219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:40.762189    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:12:40.765995 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:12:40.765974    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6788c89f75-58rcg"
Jul 28 17:13:12.223034 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:12.223000    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:12.279139 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:12.279098    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:24.133568 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:24.133527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:24.186997 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:24.186965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:27.898028 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:27.897952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:28.119862 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:28.119814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:28.213023 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:28.212935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:28.648451 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:28.648377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:28.715256 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:28.715223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:33.554842 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:33.554793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:33.656530 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:33.656492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:35.250823 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:35.250777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:38.990341 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:38.990309    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c54b6dc66-hm6mv"]
Jul 28 17:13:39.278748 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:39.278667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:39.392178 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:13:39.392141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:04.009529 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.009475    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c54b6dc66-hm6mv" podUID="e9805559-781e-4157-826b-0a2b43d72c2e" containerName="console" containerID="cri-o://8e73b5d44dd698a15c4a3be710072c5e58db02548aae3e00719639fb4504f07c" gracePeriod=15
Jul 28 17:14:04.200788 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.200755    2579 generic.go:356] "Generic (PLEG): container finished" podID="e9805559-781e-4157-826b-0a2b43d72c2e" containerID="8e73b5d44dd698a15c4a3be710072c5e58db02548aae3e00719639fb4504f07c" exitCode=0
Jul 28 17:14:04.200911 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.200821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c54b6dc66-hm6mv" event={"ID":"e9805559-781e-4157-826b-0a2b43d72c2e","Type":"ContainerDied","Data":"8e73b5d44dd698a15c4a3be710072c5e58db02548aae3e00719639fb4504f07c"}
Jul 28 17:14:04.246607 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.246588    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:14:04.349384 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-service-ca\") pod \"e9805559-781e-4157-826b-0a2b43d72c2e\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") "
Jul 28 17:14:04.349521 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-console-config\") pod \"e9805559-781e-4157-826b-0a2b43d72c2e\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") "
Jul 28 17:14:04.349521 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349446    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-trusted-ca-bundle\") pod \"e9805559-781e-4157-826b-0a2b43d72c2e\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") "
Jul 28 17:14:04.349521 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349483    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjd9b\" (UniqueName: \"kubernetes.io/projected/e9805559-781e-4157-826b-0a2b43d72c2e-kube-api-access-qjd9b\") pod \"e9805559-781e-4157-826b-0a2b43d72c2e\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") "
Jul 28 17:14:04.349521 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349513    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-oauth-config\") pod \"e9805559-781e-4157-826b-0a2b43d72c2e\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") "
Jul 28 17:14:04.349739 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-oauth-serving-cert\") pod \"e9805559-781e-4157-826b-0a2b43d72c2e\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") "
Jul 28 17:14:04.349739 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-serving-cert\") pod \"e9805559-781e-4157-826b-0a2b43d72c2e\" (UID: \"e9805559-781e-4157-826b-0a2b43d72c2e\") "
Jul 28 17:14:04.349930 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349888    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e9805559-781e-4157-826b-0a2b43d72c2e" (UID: "e9805559-781e-4157-826b-0a2b43d72c2e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:14:04.349930 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-console-config" (OuterVolumeSpecName: "console-config") pod "e9805559-781e-4157-826b-0a2b43d72c2e" (UID: "e9805559-781e-4157-826b-0a2b43d72c2e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:14:04.350076 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.349967    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e9805559-781e-4157-826b-0a2b43d72c2e" (UID: "e9805559-781e-4157-826b-0a2b43d72c2e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:14:04.350076 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.350025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-service-ca" (OuterVolumeSpecName: "service-ca") pod "e9805559-781e-4157-826b-0a2b43d72c2e" (UID: "e9805559-781e-4157-826b-0a2b43d72c2e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:14:04.351759 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.351729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e9805559-781e-4157-826b-0a2b43d72c2e" (UID: "e9805559-781e-4157-826b-0a2b43d72c2e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:14:04.351759 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.351744    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e9805559-781e-4157-826b-0a2b43d72c2e" (UID: "e9805559-781e-4157-826b-0a2b43d72c2e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:14:04.351878 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.351772    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9805559-781e-4157-826b-0a2b43d72c2e-kube-api-access-qjd9b" (OuterVolumeSpecName: "kube-api-access-qjd9b") pod "e9805559-781e-4157-826b-0a2b43d72c2e" (UID: "e9805559-781e-4157-826b-0a2b43d72c2e"). InnerVolumeSpecName "kube-api-access-qjd9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:14:04.450294 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.450262    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-oauth-serving-cert\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:14:04.450294 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.450290    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-serving-cert\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:14:04.450294 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.450300    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-service-ca\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:14:04.450522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.450310    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-console-config\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:14:04.450522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.450319    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9805559-781e-4157-826b-0a2b43d72c2e-trusted-ca-bundle\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:14:04.450522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.450328    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjd9b\" (UniqueName: \"kubernetes.io/projected/e9805559-781e-4157-826b-0a2b43d72c2e-kube-api-access-qjd9b\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:14:04.450522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:04.450337    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e9805559-781e-4157-826b-0a2b43d72c2e-console-oauth-config\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:14:05.205779 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:05.205751    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c54b6dc66-hm6mv"
Jul 28 17:14:05.206176 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:05.205755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c54b6dc66-hm6mv" event={"ID":"e9805559-781e-4157-826b-0a2b43d72c2e","Type":"ContainerDied","Data":"dc2e770b9f9edd6efb2064bb777c3fc666ae3c916f9b2f0a1f3eefea950527ed"}
Jul 28 17:14:05.206176 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:05.205901    2579 scope.go:117] "RemoveContainer" containerID="8e73b5d44dd698a15c4a3be710072c5e58db02548aae3e00719639fb4504f07c"
Jul 28 17:14:05.227204 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:05.227178    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c54b6dc66-hm6mv"]
Jul 28 17:14:05.231310 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:05.231286    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c54b6dc66-hm6mv"]
Jul 28 17:14:06.085546 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:06.085508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:06.143697 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:06.143667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:06.283901 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:14:06.283876    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9805559-781e-4157-826b-0a2b43d72c2e" path="/var/lib/kubelet/pods/e9805559-781e-4157-826b-0a2b43d72c2e/volumes"
Jul 28 17:15:06.291454 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:06.291355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:06.392015 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:06.391985    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:06.667166 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:06.667136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:06.770750 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:06.770716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:06.871805 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:06.871775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:34.048551 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:34.048510    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:34.095651 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:34.095622    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:15:34.141623 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:34.141587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:35.412017 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.411984    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"]
Jul 28 17:15:35.414280 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.412255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9805559-781e-4157-826b-0a2b43d72c2e" containerName="console"
Jul 28 17:15:35.414280 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.412265    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9805559-781e-4157-826b-0a2b43d72c2e" containerName="console"
Jul 28 17:15:35.414280 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.412318    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9805559-781e-4157-826b-0a2b43d72c2e" containerName="console"
Jul 28 17:15:35.415167 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.415151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:15:35.419006 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.418981    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 17:15:35.419243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.419228    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-p4txb\""
Jul 28 17:15:35.419773 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.419756    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 17:15:35.420113 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.420101    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 17:15:35.431459 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.431436    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"]
Jul 28 17:15:35.481569 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.481541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbdk\" (UniqueName: \"kubernetes.io/projected/f44dc480-99e6-4ed6-8ac6-2875ea7989f2-kube-api-access-kzbdk\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp\" (UID: \"f44dc480-99e6-4ed6-8ac6-2875ea7989f2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:15:35.481660 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.481581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f44dc480-99e6-4ed6-8ac6-2875ea7989f2-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp\" (UID: \"f44dc480-99e6-4ed6-8ac6-2875ea7989f2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:15:35.582496 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.582468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbdk\" (UniqueName: \"kubernetes.io/projected/f44dc480-99e6-4ed6-8ac6-2875ea7989f2-kube-api-access-kzbdk\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp\" (UID: \"f44dc480-99e6-4ed6-8ac6-2875ea7989f2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:15:35.582607 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.582509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f44dc480-99e6-4ed6-8ac6-2875ea7989f2-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp\" (UID: \"f44dc480-99e6-4ed6-8ac6-2875ea7989f2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:15:35.584700 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.584680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f44dc480-99e6-4ed6-8ac6-2875ea7989f2-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp\" (UID: \"f44dc480-99e6-4ed6-8ac6-2875ea7989f2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:15:35.591294 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.591277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbdk\" (UniqueName: \"kubernetes.io/projected/f44dc480-99e6-4ed6-8ac6-2875ea7989f2-kube-api-access-kzbdk\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp\" (UID: \"f44dc480-99e6-4ed6-8ac6-2875ea7989f2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:15:35.724497 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.724436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:15:35.850887 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.850857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"]
Jul 28 17:15:35.853960 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:15:35.853932    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44dc480_99e6_4ed6_8ac6_2875ea7989f2.slice/crio-01f2dfa79e66efeb83bdf237a3f7609f172355ea0e898fe5e7398193d90378e0 WatchSource:0}: Error finding container 01f2dfa79e66efeb83bdf237a3f7609f172355ea0e898fe5e7398193d90378e0: Status 404 returned error can't find the container with id 01f2dfa79e66efeb83bdf237a3f7609f172355ea0e898fe5e7398193d90378e0
Jul 28 17:15:35.855757 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:35.855737    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:15:36.534072 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:36.534037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp" event={"ID":"f44dc480-99e6-4ed6-8ac6-2875ea7989f2","Type":"ContainerStarted","Data":"01f2dfa79e66efeb83bdf237a3f7609f172355ea0e898fe5e7398193d90378e0"}
Jul 28 17:15:39.548501 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:39.548468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp" event={"ID":"f44dc480-99e6-4ed6-8ac6-2875ea7989f2","Type":"ContainerStarted","Data":"850b019ab304a2c038865138fb2c3147052c76cd1265d92478cafdba7a9be294"}
Jul 28 17:15:39.548889 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:39.548700    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:15:39.649611 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:39.649575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:41.755349 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:15:41.755301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:00.555310 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:00.555274    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp"
Jul 28 17:16:00.576768 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:00.576720    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-tcwpp" podStartSLOduration=22.189047821 podStartE2EDuration="25.576705555s" totalImagesPullingTime="3.387657734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:15:39.573808672 +0000 UTC m=+305.808774124" watchObservedRunningTime="2026-07-28 17:16:00.576705555 +0000 UTC m=+326.811671006"
Jul 28 17:16:57.824271 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.824238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:57.880909 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.880877    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-vp66g"]
Jul 28 17:16:57.884039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.884016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:16:57.886675 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.886650    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 17:16:57.887365 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.887345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-gq5cv\""
Jul 28 17:16:57.887490 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.887344    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:16:57.887490 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.887354    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:16:57.894026 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.894006    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-vp66g"]
Jul 28 17:16:57.908636 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.908611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:57.931814 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.931790    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-g7wgt"]
Jul 28 17:16:57.935305 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.935286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:16:57.939899 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.939881    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q52tv\""
Jul 28 17:16:57.939899 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.939890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:16:57.945322 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.945304    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-g7wgt"]
Jul 28 17:16:57.992986 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.992963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdjj\" (UniqueName: \"kubernetes.io/projected/ac0ab326-0497-42c0-8def-ab2f26849e13-kube-api-access-xrdjj\") pod \"kserve-controller-manager-cd4d44c67-vp66g\" (UID: \"ac0ab326-0497-42c0-8def-ab2f26849e13\") " pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:16:57.993109 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.992998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac0ab326-0497-42c0-8def-ab2f26849e13-cert\") pod \"kserve-controller-manager-cd4d44c67-vp66g\" (UID: \"ac0ab326-0497-42c0-8def-ab2f26849e13\") " pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:16:57.993109 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.993045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xp64\" (UniqueName: \"kubernetes.io/projected/0e178712-6841-49b8-993e-c02a373c7b17-kube-api-access-2xp64\") pod \"seaweedfs-64568bcd49-g7wgt\" (UID: \"0e178712-6841-49b8-993e-c02a373c7b17\") " pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:16:57.993109 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:57.993100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0e178712-6841-49b8-993e-c02a373c7b17-data\") pod \"seaweedfs-64568bcd49-g7wgt\" (UID: \"0e178712-6841-49b8-993e-c02a373c7b17\") " pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:16:58.013217 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.013189    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:58.093665 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.093606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xp64\" (UniqueName: \"kubernetes.io/projected/0e178712-6841-49b8-993e-c02a373c7b17-kube-api-access-2xp64\") pod \"seaweedfs-64568bcd49-g7wgt\" (UID: \"0e178712-6841-49b8-993e-c02a373c7b17\") " pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:16:58.093665 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.093644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0e178712-6841-49b8-993e-c02a373c7b17-data\") pod \"seaweedfs-64568bcd49-g7wgt\" (UID: \"0e178712-6841-49b8-993e-c02a373c7b17\") " pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:16:58.093790 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.093669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdjj\" (UniqueName: \"kubernetes.io/projected/ac0ab326-0497-42c0-8def-ab2f26849e13-kube-api-access-xrdjj\") pod \"kserve-controller-manager-cd4d44c67-vp66g\" (UID: \"ac0ab326-0497-42c0-8def-ab2f26849e13\") " pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:16:58.093826 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.093789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac0ab326-0497-42c0-8def-ab2f26849e13-cert\") pod \"kserve-controller-manager-cd4d44c67-vp66g\" (UID: \"ac0ab326-0497-42c0-8def-ab2f26849e13\") " pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:16:58.094001 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.093983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0e178712-6841-49b8-993e-c02a373c7b17-data\") pod \"seaweedfs-64568bcd49-g7wgt\" (UID: \"0e178712-6841-49b8-993e-c02a373c7b17\") " pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:16:58.096187 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.096167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac0ab326-0497-42c0-8def-ab2f26849e13-cert\") pod \"kserve-controller-manager-cd4d44c67-vp66g\" (UID: \"ac0ab326-0497-42c0-8def-ab2f26849e13\") " pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:16:58.101836 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.101810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xp64\" (UniqueName: \"kubernetes.io/projected/0e178712-6841-49b8-993e-c02a373c7b17-kube-api-access-2xp64\") pod \"seaweedfs-64568bcd49-g7wgt\" (UID: \"0e178712-6841-49b8-993e-c02a373c7b17\") " pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:16:58.101930 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.101895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdjj\" (UniqueName: \"kubernetes.io/projected/ac0ab326-0497-42c0-8def-ab2f26849e13-kube-api-access-xrdjj\") pod \"kserve-controller-manager-cd4d44c67-vp66g\" (UID: \"ac0ab326-0497-42c0-8def-ab2f26849e13\") " pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:16:58.113941 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.113920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:58.196215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.196191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:16:58.244813 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.244787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:16:58.332356 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.332326    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-vp66g"]
Jul 28 17:16:58.334441 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:16:58.334393    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0ab326_0497_42c0_8def_ab2f26849e13.slice/crio-4b742ede5f2105f122bbaffd4d256b8d53d66796ed3f277f9b4a01ab9264d964 WatchSource:0}: Error finding container 4b742ede5f2105f122bbaffd4d256b8d53d66796ed3f277f9b4a01ab9264d964: Status 404 returned error can't find the container with id 4b742ede5f2105f122bbaffd4d256b8d53d66796ed3f277f9b4a01ab9264d964
Jul 28 17:16:58.379360 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.379337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-g7wgt"]
Jul 28 17:16:58.381488 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:16:58.381466    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e178712_6841_49b8_993e_c02a373c7b17.slice/crio-e6099ab3ec741923ae20f89b374f2cd25e1b36c8441311a85ca4b9ac10e0df62 WatchSource:0}: Error finding container e6099ab3ec741923ae20f89b374f2cd25e1b36c8441311a85ca4b9ac10e0df62: Status 404 returned error can't find the container with id e6099ab3ec741923ae20f89b374f2cd25e1b36c8441311a85ca4b9ac10e0df62
Jul 28 17:16:58.448967 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.448932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:58.543839 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.543809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:58.852938 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.852904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-g7wgt" event={"ID":"0e178712-6841-49b8-993e-c02a373c7b17","Type":"ContainerStarted","Data":"e6099ab3ec741923ae20f89b374f2cd25e1b36c8441311a85ca4b9ac10e0df62"}
Jul 28 17:16:58.854057 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:16:58.854034    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g" event={"ID":"ac0ab326-0497-42c0-8def-ab2f26849e13","Type":"ContainerStarted","Data":"4b742ede5f2105f122bbaffd4d256b8d53d66796ed3f277f9b4a01ab9264d964"}
Jul 28 17:17:02.876341 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:02.876304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-g7wgt" event={"ID":"0e178712-6841-49b8-993e-c02a373c7b17","Type":"ContainerStarted","Data":"35f05f8a1f7d6ce92d78049a94c53e24102051780df30daaa498cba74eb3ac62"}
Jul 28 17:17:02.876755 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:02.876432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:17:02.877839 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:02.877819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g" event={"ID":"ac0ab326-0497-42c0-8def-ab2f26849e13","Type":"ContainerStarted","Data":"711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800"}
Jul 28 17:17:02.877898 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:02.877869    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:17:02.894222 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:02.894174    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-g7wgt" podStartSLOduration=2.223080222 podStartE2EDuration="5.8941593s" totalImagesPullingTime="3.671079078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:02.893551646 +0000 UTC m=+389.128517101" watchObservedRunningTime="2026-07-28 17:17:02.8941593 +0000 UTC m=+389.129124753"
Jul 28 17:17:02.912864 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:02.912816    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g" podStartSLOduration=2.281826018 podStartE2EDuration="5.912803182s" totalImagesPullingTime="3.630977164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:02.911969537 +0000 UTC m=+389.146934990" watchObservedRunningTime="2026-07-28 17:17:02.912803182 +0000 UTC m=+389.147768633"
Jul 28 17:17:08.883091 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:08.883060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-g7wgt"
Jul 28 17:17:10.191112 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:10.191077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:22.260006 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.259965    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-657d64dcd5-52p67"]
Jul 28 17:17:22.264703 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.264681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.267160 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.267133    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:17:22.267280 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.267134    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:17:22.267917 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.267898    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l68jz\""
Jul 28 17:17:22.268015 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.267915    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:17:22.268110 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.268090    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:17:22.268217 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.268200    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:17:22.272165 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.272143    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:17:22.275066 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.275046    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-657d64dcd5-52p67"]
Jul 28 17:17:22.327709 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.327675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:22.379517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.379490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-service-ca\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.379643 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.379521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-console-serving-cert\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.379643 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.379560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjgt\" (UniqueName: \"kubernetes.io/projected/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-kube-api-access-qnjgt\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.379643 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.379588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-trusted-ca-bundle\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.379643 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.379610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-console-oauth-config\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.379643 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.379629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-oauth-serving-cert\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.379803 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.379654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-console-config\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.391236 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.391206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:22.464328 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.464292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:22.480162 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.480133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-console-oauth-config\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.480275 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.480165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-oauth-serving-cert\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.480275 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.480184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-console-config\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.480275 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.480252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-service-ca\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.480419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.480275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-console-serving-cert\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.480419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.480318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjgt\" (UniqueName: \"kubernetes.io/projected/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-kube-api-access-qnjgt\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.480419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.480339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-trusted-ca-bundle\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.481040 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.481008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-oauth-serving-cert\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.481137 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.481053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-service-ca\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.481137 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.481071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-console-config\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.481229 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.481213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-trusted-ca-bundle\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.483022 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.482953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-console-oauth-config\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.487683 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.483194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-console-serving-cert\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.490112 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.490084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjgt\" (UniqueName: \"kubernetes.io/projected/d117fd2f-5e97-4b3c-ae7f-cd8f00b00638-kube-api-access-qnjgt\") pod \"console-657d64dcd5-52p67\" (UID: \"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638\") " pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.565716 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.565682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:22.574692 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.574670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:22.706674 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.706649    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-657d64dcd5-52p67"]
Jul 28 17:17:22.708529 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:17:22.708502    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd117fd2f_5e97_4b3c_ae7f_cd8f00b00638.slice/crio-3091592174bc8a75e4f5433f77ff8c732bcb53df2fbfbc0b6b8ba3c6a0d1c5c8 WatchSource:0}: Error finding container 3091592174bc8a75e4f5433f77ff8c732bcb53df2fbfbc0b6b8ba3c6a0d1c5c8: Status 404 returned error can't find the container with id 3091592174bc8a75e4f5433f77ff8c732bcb53df2fbfbc0b6b8ba3c6a0d1c5c8
Jul 28 17:17:22.956762 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.956687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657d64dcd5-52p67" event={"ID":"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638","Type":"ContainerStarted","Data":"d29b7735cbca76b9d1d4933366656376e586476acb9a9fc3b4865ed716b71194"}
Jul 28 17:17:22.956762 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.956732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657d64dcd5-52p67" event={"ID":"d117fd2f-5e97-4b3c-ae7f-cd8f00b00638","Type":"ContainerStarted","Data":"3091592174bc8a75e4f5433f77ff8c732bcb53df2fbfbc0b6b8ba3c6a0d1c5c8"}
Jul 28 17:17:22.985455 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:22.985393    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-657d64dcd5-52p67" podStartSLOduration=0.985378219 podStartE2EDuration="985.378219ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:22.984477677 +0000 UTC m=+409.219443131" watchObservedRunningTime="2026-07-28 17:17:22.985378219 +0000 UTC m=+409.220343671"
Jul 28 17:17:23.060337 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:23.060280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:32.575274 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:32.575234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:32.575777 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:32.575304    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:32.579996 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:32.579974    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:33.002507 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:33.002421    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-657d64dcd5-52p67"
Jul 28 17:17:33.171935 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:33.171897    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:33.272245 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:33.272168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:33.374161 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:33.374129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:33.887004 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:33.886975    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:17:40.771893 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:40.771856    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:40.873466 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:40.873434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:52.046322 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:52.046285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:52.855494 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:52.855458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:53.919626 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:53.919592    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"]
Jul 28 17:17:53.926289 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:53.926269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"
Jul 28 17:17:53.929394 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:53.929372    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7vvt4\""
Jul 28 17:17:53.929687 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:53.929669    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:53.931335 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:53.931315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:53.939743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:53.939714    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"]
Jul 28 17:17:54.014782 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:54.014752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b599567-dc6d-4744-8e28-1502ebf6c9a1-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-v2bx6\" (UID: \"0b599567-dc6d-4744-8e28-1502ebf6c9a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"
Jul 28 17:17:54.014925 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:54.014809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgl2s\" (UniqueName: \"kubernetes.io/projected/0b599567-dc6d-4744-8e28-1502ebf6c9a1-kube-api-access-sgl2s\") pod \"cert-manager-operator-controller-manager-684f79c9b9-v2bx6\" (UID: \"0b599567-dc6d-4744-8e28-1502ebf6c9a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"
Jul 28 17:17:54.115137 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:54.115107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b599567-dc6d-4744-8e28-1502ebf6c9a1-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-v2bx6\" (UID: \"0b599567-dc6d-4744-8e28-1502ebf6c9a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"
Jul 28 17:17:54.115269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:54.115160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgl2s\" (UniqueName: \"kubernetes.io/projected/0b599567-dc6d-4744-8e28-1502ebf6c9a1-kube-api-access-sgl2s\") pod \"cert-manager-operator-controller-manager-684f79c9b9-v2bx6\" (UID: \"0b599567-dc6d-4744-8e28-1502ebf6c9a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"
Jul 28 17:17:54.115498 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:54.115479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b599567-dc6d-4744-8e28-1502ebf6c9a1-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-v2bx6\" (UID: \"0b599567-dc6d-4744-8e28-1502ebf6c9a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"
Jul 28 17:17:54.123628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:54.123602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgl2s\" (UniqueName: \"kubernetes.io/projected/0b599567-dc6d-4744-8e28-1502ebf6c9a1-kube-api-access-sgl2s\") pod \"cert-manager-operator-controller-manager-684f79c9b9-v2bx6\" (UID: \"0b599567-dc6d-4744-8e28-1502ebf6c9a1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"
Jul 28 17:17:54.234911 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:54.234839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"
Jul 28 17:17:54.366369 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:54.366339    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6"]
Jul 28 17:17:54.369139 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:17:54.369111    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b599567_dc6d_4744_8e28_1502ebf6c9a1.slice/crio-df2bf0ee9be7d818192e9be181747c3a94636f3411c506d430ca3655211ef871 WatchSource:0}: Error finding container df2bf0ee9be7d818192e9be181747c3a94636f3411c506d430ca3655211ef871: Status 404 returned error can't find the container with id df2bf0ee9be7d818192e9be181747c3a94636f3411c506d430ca3655211ef871
Jul 28 17:17:55.089311 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:55.089270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6" event={"ID":"0b599567-dc6d-4744-8e28-1502ebf6c9a1","Type":"ContainerStarted","Data":"df2bf0ee9be7d818192e9be181747c3a94636f3411c506d430ca3655211ef871"}
Jul 28 17:17:55.751592 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:55.751555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:55.810681 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:55.810642    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:57.100883 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:57.100846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6" event={"ID":"0b599567-dc6d-4744-8e28-1502ebf6c9a1","Type":"ContainerStarted","Data":"81271917dacd45c39bc2c9c9dcae38f1e0f032b5567c16c2baf5c02e092abb74"}
Jul 28 17:17:57.124211 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:57.124158    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-v2bx6" podStartSLOduration=1.583661054 podStartE2EDuration="4.124137937s" totalImagesPullingTime="2.540476883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:57.12092011 +0000 UTC m=+443.355885586" watchObservedRunningTime="2026-07-28 17:17:57.124137937 +0000 UTC m=+443.359103376"
Jul 28 17:17:57.822408 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:57.822358    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:57.946116 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:57.946085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:58.045583 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:17:58.045558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:00.213760 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.213724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:00.446490 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.446455    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:00.548358 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.548322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:00.647965 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.647934    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:00.730120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.730093    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"]
Jul 28 17:18:00.733313 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.733296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"
Jul 28 17:18:00.735907 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.735881    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 17:18:00.736025 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.735971    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k9hkx\""
Jul 28 17:18:00.736966 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.736947    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 17:18:00.743131 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.743105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"]
Jul 28 17:18:00.870100 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.870039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4krv\" (UniqueName: \"kubernetes.io/projected/c66f26f0-b318-4da9-a52a-e48e32e5e688-kube-api-access-n4krv\") pod \"cert-manager-cainjector-559cdd8c87-vsg4n\" (UID: \"c66f26f0-b318-4da9-a52a-e48e32e5e688\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"
Jul 28 17:18:00.870100 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.870075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c66f26f0-b318-4da9-a52a-e48e32e5e688-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vsg4n\" (UID: \"c66f26f0-b318-4da9-a52a-e48e32e5e688\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"
Jul 28 17:18:00.970565 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.970540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4krv\" (UniqueName: \"kubernetes.io/projected/c66f26f0-b318-4da9-a52a-e48e32e5e688-kube-api-access-n4krv\") pod \"cert-manager-cainjector-559cdd8c87-vsg4n\" (UID: \"c66f26f0-b318-4da9-a52a-e48e32e5e688\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"
Jul 28 17:18:00.970674 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.970582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c66f26f0-b318-4da9-a52a-e48e32e5e688-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vsg4n\" (UID: \"c66f26f0-b318-4da9-a52a-e48e32e5e688\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"
Jul 28 17:18:00.983792 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.983767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4krv\" (UniqueName: \"kubernetes.io/projected/c66f26f0-b318-4da9-a52a-e48e32e5e688-kube-api-access-n4krv\") pod \"cert-manager-cainjector-559cdd8c87-vsg4n\" (UID: \"c66f26f0-b318-4da9-a52a-e48e32e5e688\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"
Jul 28 17:18:00.984280 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:00.984262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c66f26f0-b318-4da9-a52a-e48e32e5e688-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vsg4n\" (UID: \"c66f26f0-b318-4da9-a52a-e48e32e5e688\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"
Jul 28 17:18:01.042342 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:01.042319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"
Jul 28 17:18:01.173894 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:01.173872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n"]
Jul 28 17:18:01.175939 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:18:01.175904    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc66f26f0_b318_4da9_a52a_e48e32e5e688.slice/crio-0455f1f537b715888d4a1d6bf2400903cba927715163d036caa12b610af7a739 WatchSource:0}: Error finding container 0455f1f537b715888d4a1d6bf2400903cba927715163d036caa12b610af7a739: Status 404 returned error can't find the container with id 0455f1f537b715888d4a1d6bf2400903cba927715163d036caa12b610af7a739
Jul 28 17:18:02.122624 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:02.122586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n" event={"ID":"c66f26f0-b318-4da9-a52a-e48e32e5e688","Type":"ContainerStarted","Data":"0455f1f537b715888d4a1d6bf2400903cba927715163d036caa12b610af7a739"}
Jul 28 17:18:02.193978 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:02.193942    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:02.296810 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:02.296779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:02.394591 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:02.394502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:05.136882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:05.136844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n" event={"ID":"c66f26f0-b318-4da9-a52a-e48e32e5e688","Type":"ContainerStarted","Data":"e209eccec4ac63adbd564e8062c6cbff87ebec5d19ac1505550d18479982e943"}
Jul 28 17:18:05.155386 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:05.155340    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vsg4n" podStartSLOduration=2.23329831 podStartE2EDuration="5.155325243s" totalImagesPullingTime="2.922026933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:05.154038687 +0000 UTC m=+451.389004195" watchObservedRunningTime="2026-07-28 17:18:05.155325243 +0000 UTC m=+451.390290694"
Jul 28 17:18:07.824533 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:07.824503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:09.154782 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.154745    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4nnbn"]
Jul 28 17:18:09.158205 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.158183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4nnbn"
Jul 28 17:18:09.160612 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.160592    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mb5n8\""
Jul 28 17:18:09.167556 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.167533    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4nnbn"]
Jul 28 17:18:09.238509 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.238486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxj8\" (UniqueName: \"kubernetes.io/projected/1cbfe0a4-1005-453f-8806-566288c2035d-kube-api-access-6hxj8\") pod \"cert-manager-55b4d954d5-4nnbn\" (UID: \"1cbfe0a4-1005-453f-8806-566288c2035d\") " pod="cert-manager/cert-manager-55b4d954d5-4nnbn"
Jul 28 17:18:09.238616 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.238517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cbfe0a4-1005-453f-8806-566288c2035d-bound-sa-token\") pod \"cert-manager-55b4d954d5-4nnbn\" (UID: \"1cbfe0a4-1005-453f-8806-566288c2035d\") " pod="cert-manager/cert-manager-55b4d954d5-4nnbn"
Jul 28 17:18:09.339861 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.339830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxj8\" (UniqueName: \"kubernetes.io/projected/1cbfe0a4-1005-453f-8806-566288c2035d-kube-api-access-6hxj8\") pod \"cert-manager-55b4d954d5-4nnbn\" (UID: \"1cbfe0a4-1005-453f-8806-566288c2035d\") " pod="cert-manager/cert-manager-55b4d954d5-4nnbn"
Jul 28 17:18:09.339972 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.339867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cbfe0a4-1005-453f-8806-566288c2035d-bound-sa-token\") pod \"cert-manager-55b4d954d5-4nnbn\" (UID: \"1cbfe0a4-1005-453f-8806-566288c2035d\") " pod="cert-manager/cert-manager-55b4d954d5-4nnbn"
Jul 28 17:18:09.348680 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.348653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cbfe0a4-1005-453f-8806-566288c2035d-bound-sa-token\") pod \"cert-manager-55b4d954d5-4nnbn\" (UID: \"1cbfe0a4-1005-453f-8806-566288c2035d\") " pod="cert-manager/cert-manager-55b4d954d5-4nnbn"
Jul 28 17:18:09.348876 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.348858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxj8\" (UniqueName: \"kubernetes.io/projected/1cbfe0a4-1005-453f-8806-566288c2035d-kube-api-access-6hxj8\") pod \"cert-manager-55b4d954d5-4nnbn\" (UID: \"1cbfe0a4-1005-453f-8806-566288c2035d\") " pod="cert-manager/cert-manager-55b4d954d5-4nnbn"
Jul 28 17:18:09.468221 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.468155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4nnbn"
Jul 28 17:18:09.608901 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:09.608760    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4nnbn"]
Jul 28 17:18:09.611505 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:18:09.611462    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cbfe0a4_1005_453f_8806_566288c2035d.slice/crio-c73a58cea6fca116cbfd1583aea3fae1a1e6b8bc9af309884b4766eb81e9e054 WatchSource:0}: Error finding container c73a58cea6fca116cbfd1583aea3fae1a1e6b8bc9af309884b4766eb81e9e054: Status 404 returned error can't find the container with id c73a58cea6fca116cbfd1583aea3fae1a1e6b8bc9af309884b4766eb81e9e054
Jul 28 17:18:10.159569 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:10.159536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4nnbn" event={"ID":"1cbfe0a4-1005-453f-8806-566288c2035d","Type":"ContainerStarted","Data":"18aaaf5393163cb9da7aeaa925aa58da23457d6c83e067438f2c7036dc02b328"}
Jul 28 17:18:10.159569 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:10.159576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4nnbn" event={"ID":"1cbfe0a4-1005-453f-8806-566288c2035d","Type":"ContainerStarted","Data":"c73a58cea6fca116cbfd1583aea3fae1a1e6b8bc9af309884b4766eb81e9e054"}
Jul 28 17:18:10.176983 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:10.176934    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4nnbn" podStartSLOduration=1.176918563 podStartE2EDuration="1.176918563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:10.17594047 +0000 UTC m=+456.410905922" watchObservedRunningTime="2026-07-28 17:18:10.176918563 +0000 UTC m=+456.411884015"
Jul 28 17:18:14.152586 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:14.152549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:14.304351 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:14.304314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.422047 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:23.422010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:23.488069 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:23.488028    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:26.180513 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:26.180467    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:30.571204 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.571167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:30.632020 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.631993    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"]
Jul 28 17:18:30.635232 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.635215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.637787 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.637771    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 17:18:30.638332 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.638315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 17:18:30.638785 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.638769    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:18:30.638838 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.638786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 17:18:30.638838 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.638807    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mfs9z\""
Jul 28 17:18:30.639015 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.638994    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 17:18:30.649346 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.649323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"]
Jul 28 17:18:30.677729 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.677702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:30.708672 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.708650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/775fa999-365a-46d5-a31d-17ddbd90f20a-manager-config\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.708740 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.708678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/775fa999-365a-46d5-a31d-17ddbd90f20a-cert\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.708740 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.708700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/775fa999-365a-46d5-a31d-17ddbd90f20a-metrics-cert\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.708740 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.708717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcsdg\" (UniqueName: \"kubernetes.io/projected/775fa999-365a-46d5-a31d-17ddbd90f20a-kube-api-access-pcsdg\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.809248 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.809216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/775fa999-365a-46d5-a31d-17ddbd90f20a-manager-config\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.809248 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.809251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/775fa999-365a-46d5-a31d-17ddbd90f20a-cert\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.809467 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.809274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/775fa999-365a-46d5-a31d-17ddbd90f20a-metrics-cert\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.809467 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.809293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcsdg\" (UniqueName: \"kubernetes.io/projected/775fa999-365a-46d5-a31d-17ddbd90f20a-kube-api-access-pcsdg\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.809975 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.809954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/775fa999-365a-46d5-a31d-17ddbd90f20a-manager-config\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.812135 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.812102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/775fa999-365a-46d5-a31d-17ddbd90f20a-cert\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.812135 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.812117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/775fa999-365a-46d5-a31d-17ddbd90f20a-metrics-cert\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.820064 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.820043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcsdg\" (UniqueName: \"kubernetes.io/projected/775fa999-365a-46d5-a31d-17ddbd90f20a-kube-api-access-pcsdg\") pod \"lws-controller-manager-659b67c655-7r4r7\" (UID: \"775fa999-365a-46d5-a31d-17ddbd90f20a\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:30.937791 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.937665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:30.944573 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:30.944548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:31.034092 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:31.034059    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:31.099004 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:31.098977    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"]
Jul 28 17:18:31.101585 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:18:31.101546    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775fa999_365a_46d5_a31d_17ddbd90f20a.slice/crio-b315386e2186d2659323e603f8a9547ffa1cb6fdb39f14cd6049093e6e97dd59 WatchSource:0}: Error finding container b315386e2186d2659323e603f8a9547ffa1cb6fdb39f14cd6049093e6e97dd59: Status 404 returned error can't find the container with id b315386e2186d2659323e603f8a9547ffa1cb6fdb39f14cd6049093e6e97dd59
Jul 28 17:18:31.246458 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:31.246368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7" event={"ID":"775fa999-365a-46d5-a31d-17ddbd90f20a","Type":"ContainerStarted","Data":"b315386e2186d2659323e603f8a9547ffa1cb6fdb39f14cd6049093e6e97dd59"}
Jul 28 17:18:32.110097 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:32.110060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:34.208977 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:34.208943    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:34.604753 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:34.604714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:34.667724 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:34.667688    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:38.566023 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:38.565991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:38.703733 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:38.703640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:42.830941 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:42.830907    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:43.299433 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:43.299380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7" event={"ID":"775fa999-365a-46d5-a31d-17ddbd90f20a","Type":"ContainerStarted","Data":"7593430212d95afa36d9654b75f30e4d1c88e9180dc893c9c6e11deb85d2dd6c"}
Jul 28 17:18:43.299588 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:43.299452    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:18:43.318488 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:43.318436    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7" podStartSLOduration=2.169868955 podStartE2EDuration="13.318421366s" totalImagesPullingTime="11.148552411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:43.316568625 +0000 UTC m=+489.551534112" watchObservedRunningTime="2026-07-28 17:18:43.318421366 +0000 UTC m=+489.553386818"
Jul 28 17:18:45.216509 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:45.216473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:45.390125 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:45.390092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:54.304977 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:18:54.304948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-659b67c655-7r4r7"
Jul 28 17:19:04.266998 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:04.266955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:05.060024 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:05.059988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:05.323583 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:05.323502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:05.932449 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:05.932394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:09.098555 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:09.098516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:09.189027 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:09.188991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:09.482565 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:09.482489    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:11.087601 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:11.087567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:11.477168 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:11.477092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:12.131288 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:12.131251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:14.088228 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:14.088188    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:14.315119 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:14.315089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:14.418541 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:14.418462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:15.746016 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.745976    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"]
Jul 28 17:19:15.748431 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.748374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:15.754467 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.754447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"
Jul 28 17:19:15.757773 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.757745    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 17:19:15.757773 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.757764    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sxdwd\""
Jul 28 17:19:15.757969 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.757951    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 17:19:15.758030 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.757955    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 17:19:15.764697 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.764675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"]
Jul 28 17:19:15.853243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.853211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdt2g\" (UniqueName: \"kubernetes.io/projected/d736d71a-cd17-4573-8b79-365d3c6d9b85-kube-api-access-sdt2g\") pod \"dns-operator-controller-manager-ccbdf686-hsv2h\" (UID: \"d736d71a-cd17-4573-8b79-365d3c6d9b85\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"
Jul 28 17:19:15.954611 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.954575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdt2g\" (UniqueName: \"kubernetes.io/projected/d736d71a-cd17-4573-8b79-365d3c6d9b85-kube-api-access-sdt2g\") pod \"dns-operator-controller-manager-ccbdf686-hsv2h\" (UID: \"d736d71a-cd17-4573-8b79-365d3c6d9b85\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"
Jul 28 17:19:15.964381 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:15.964352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdt2g\" (UniqueName: \"kubernetes.io/projected/d736d71a-cd17-4573-8b79-365d3c6d9b85-kube-api-access-sdt2g\") pod \"dns-operator-controller-manager-ccbdf686-hsv2h\" (UID: \"d736d71a-cd17-4573-8b79-365d3c6d9b85\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"
Jul 28 17:19:16.065271 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.065241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"
Jul 28 17:19:16.200557 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.200521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"]
Jul 28 17:19:16.204528 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:19:16.204498    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd736d71a_cd17_4573_8b79_365d3c6d9b85.slice/crio-f9a911ce88df6feffe5bf4c6da6f8ba61d5ab8fafa29d66b6f20427e2c71769f WatchSource:0}: Error finding container f9a911ce88df6feffe5bf4c6da6f8ba61d5ab8fafa29d66b6f20427e2c71769f: Status 404 returned error can't find the container with id f9a911ce88df6feffe5bf4c6da6f8ba61d5ab8fafa29d66b6f20427e2c71769f
Jul 28 17:19:16.391286 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.391192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:16.401066 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.401033    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"]
Jul 28 17:19:16.406134 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.406108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:16.408267 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.408239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"]
Jul 28 17:19:16.408752 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.408732    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hg4cd\""
Jul 28 17:19:16.408841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.408755    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 17:19:16.408841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.408741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 17:19:16.442962 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.442935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h" event={"ID":"d736d71a-cd17-4573-8b79-365d3c6d9b85","Type":"ContainerStarted","Data":"f9a911ce88df6feffe5bf4c6da6f8ba61d5ab8fafa29d66b6f20427e2c71769f"}
Jul 28 17:19:16.559204 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.559181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf58333-3a4d-47ee-bd45-c944a60bc013-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:16.559297 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.559228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxv7\" (UniqueName: \"kubernetes.io/projected/faf58333-3a4d-47ee-bd45-c944a60bc013-kube-api-access-fdxv7\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:16.559337 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.559299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/faf58333-3a4d-47ee-bd45-c944a60bc013-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:16.660507 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.660451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxv7\" (UniqueName: \"kubernetes.io/projected/faf58333-3a4d-47ee-bd45-c944a60bc013-kube-api-access-fdxv7\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:16.660593 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.660505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/faf58333-3a4d-47ee-bd45-c944a60bc013-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:16.660593 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.660540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf58333-3a4d-47ee-bd45-c944a60bc013-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:16.660673 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:19:16.660658    2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 17:19:16.660734 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:19:16.660724    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faf58333-3a4d-47ee-bd45-c944a60bc013-plugin-serving-cert podName:faf58333-3a4d-47ee-bd45-c944a60bc013 nodeName:}" failed. No retries permitted until 2026-07-28 17:19:17.160704208 +0000 UTC m=+523.395669644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/faf58333-3a4d-47ee-bd45-c944a60bc013-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-ffwbm" (UID: "faf58333-3a4d-47ee-bd45-c944a60bc013") : secret "plugin-serving-cert" not found
Jul 28 17:19:16.661133 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.661116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/faf58333-3a4d-47ee-bd45-c944a60bc013-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:16.672419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:16.672372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxv7\" (UniqueName: \"kubernetes.io/projected/faf58333-3a4d-47ee-bd45-c944a60bc013-kube-api-access-fdxv7\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:17.165787 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:17.165754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf58333-3a4d-47ee-bd45-c944a60bc013-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:17.166156 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:19:17.165929    2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 17:19:17.166156 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:19:17.166029    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faf58333-3a4d-47ee-bd45-c944a60bc013-plugin-serving-cert podName:faf58333-3a4d-47ee-bd45-c944a60bc013 nodeName:}" failed. No retries permitted until 2026-07-28 17:19:18.16600927 +0000 UTC m=+524.400974716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/faf58333-3a4d-47ee-bd45-c944a60bc013-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-ffwbm" (UID: "faf58333-3a4d-47ee-bd45-c944a60bc013") : secret "plugin-serving-cert" not found
Jul 28 17:19:17.289500 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:17.289467    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:18.172541 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:18.172507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf58333-3a4d-47ee-bd45-c944a60bc013-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:18.175003 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:18.174979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf58333-3a4d-47ee-bd45-c944a60bc013-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ffwbm\" (UID: \"faf58333-3a4d-47ee-bd45-c944a60bc013\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:18.216489 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:18.216464    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"
Jul 28 17:19:18.345556 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:18.345528    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm"]
Jul 28 17:19:18.347360 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:19:18.347332    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaf58333_3a4d_47ee_bd45_c944a60bc013.slice/crio-2c2ff64404541064c9c96311325d4263e1fcd94b8309ae5a2514d6e0604bdc10 WatchSource:0}: Error finding container 2c2ff64404541064c9c96311325d4263e1fcd94b8309ae5a2514d6e0604bdc10: Status 404 returned error can't find the container with id 2c2ff64404541064c9c96311325d4263e1fcd94b8309ae5a2514d6e0604bdc10
Jul 28 17:19:18.452494 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:18.452435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm" event={"ID":"faf58333-3a4d-47ee-bd45-c944a60bc013","Type":"ContainerStarted","Data":"2c2ff64404541064c9c96311325d4263e1fcd94b8309ae5a2514d6e0604bdc10"}
Jul 28 17:19:19.678941 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:19.678905    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:19.745039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:19.745004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:20.067267 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:20.067223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:20.130677 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:20.130643    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:20.469180 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:20.469104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:20.569705 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:20.569678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:21.466827 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:21.466793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h" event={"ID":"d736d71a-cd17-4573-8b79-365d3c6d9b85","Type":"ContainerStarted","Data":"5dd0f2ea057b8ec0e72fec130bd923f852a1a082875ad987511ff4105a90284c"}
Jul 28 17:19:21.467181 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:21.466937    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"
Jul 28 17:19:21.489387 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:21.489341    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h" podStartSLOduration=2.185522748 podStartE2EDuration="6.489324369s" totalImagesPullingTime="4.303801621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:21.486696213 +0000 UTC m=+527.721661665" watchObservedRunningTime="2026-07-28 17:19:21.489324369 +0000 UTC m=+527.724289821"
Jul 28 17:19:26.709713 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:26.709612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:26.810211 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:26.810124    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:27.498781 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:27.498700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm" event={"ID":"faf58333-3a4d-47ee-bd45-c944a60bc013","Type":"ContainerStarted","Data":"3e9b05f8234486685fca0757fb6e2b5366a7edd013fa5b5357fa42dc33cf6fc6"}
Jul 28 17:19:27.518103 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:27.518058    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ffwbm" podStartSLOduration=2.7968854260000002 podStartE2EDuration="11.518044157s" totalImagesPullingTime="8.721158731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:27.516494436 +0000 UTC m=+533.751459909" watchObservedRunningTime="2026-07-28 17:19:27.518044157 +0000 UTC m=+533.753009610"
Jul 28 17:19:31.497670 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:31.497630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:31.596337 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:31.596307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:32.265460 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:32.265431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:32.365320 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:32.365288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:32.473803 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:32.473775    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-hsv2h"
Jul 28 17:19:33.959342 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:33.959312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:34.021382 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:34.021346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:40.697219 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:40.697180    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:40.757338 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:40.757288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:41.085349 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:41.085315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:41.147957 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:41.147930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:41.499002 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:41.498913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:41.561221 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:41.561194    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:48.103696 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:48.103657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:48.206334 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:48.206305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:52.896616 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:52.896580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:52.997243 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:52.997213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:55.350847 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:55.350796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:55.451021 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:55.450983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:57.646829 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:57.646792    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:57.760087 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:57.760057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:57.820615 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:57.820585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:58.473812 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.473777    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-74wmk"]
Jul 28 17:19:58.477056 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.477034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-74wmk"
Jul 28 17:19:58.479561 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.479538    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vxhcz\""
Jul 28 17:19:58.488884 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.488860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-74wmk"]
Jul 28 17:19:58.519430 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.516262    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:58.596313 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.596273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4kg\" (UniqueName: \"kubernetes.io/projected/7cf64800-4ce3-43a1-989c-c237e33c1dd8-kube-api-access-kk4kg\") pod \"authorino-b54df94b4-74wmk\" (UID: \"7cf64800-4ce3-43a1-989c-c237e33c1dd8\") " pod="kuadrant-system/authorino-b54df94b4-74wmk"
Jul 28 17:19:58.619611 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.619556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:58.627026 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.627004    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-stc92"]
Jul 28 17:19:58.630286 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.630270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-stc92"
Jul 28 17:19:58.639656 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.639635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-stc92"]
Jul 28 17:19:58.697720 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.697691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz79d\" (UniqueName: \"kubernetes.io/projected/6f367907-cc2a-4cd5-92da-c18390cff656-kube-api-access-gz79d\") pod \"authorino-c548b5d7d-stc92\" (UID: \"6f367907-cc2a-4cd5-92da-c18390cff656\") " pod="kuadrant-system/authorino-c548b5d7d-stc92"
Jul 28 17:19:58.698076 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.697755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4kg\" (UniqueName: \"kubernetes.io/projected/7cf64800-4ce3-43a1-989c-c237e33c1dd8-kube-api-access-kk4kg\") pod \"authorino-b54df94b4-74wmk\" (UID: \"7cf64800-4ce3-43a1-989c-c237e33c1dd8\") " pod="kuadrant-system/authorino-b54df94b4-74wmk"
Jul 28 17:19:58.705986 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.705962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4kg\" (UniqueName: \"kubernetes.io/projected/7cf64800-4ce3-43a1-989c-c237e33c1dd8-kube-api-access-kk4kg\") pod \"authorino-b54df94b4-74wmk\" (UID: \"7cf64800-4ce3-43a1-989c-c237e33c1dd8\") " pod="kuadrant-system/authorino-b54df94b4-74wmk"
Jul 28 17:19:58.786621 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.786557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-74wmk"
Jul 28 17:19:58.798638 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.798613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz79d\" (UniqueName: \"kubernetes.io/projected/6f367907-cc2a-4cd5-92da-c18390cff656-kube-api-access-gz79d\") pod \"authorino-c548b5d7d-stc92\" (UID: \"6f367907-cc2a-4cd5-92da-c18390cff656\") " pod="kuadrant-system/authorino-c548b5d7d-stc92"
Jul 28 17:19:58.806890 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.806860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz79d\" (UniqueName: \"kubernetes.io/projected/6f367907-cc2a-4cd5-92da-c18390cff656-kube-api-access-gz79d\") pod \"authorino-c548b5d7d-stc92\" (UID: \"6f367907-cc2a-4cd5-92da-c18390cff656\") " pod="kuadrant-system/authorino-c548b5d7d-stc92"
Jul 28 17:19:58.917074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.917050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-74wmk"]
Jul 28 17:19:58.919499 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:19:58.919473    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cf64800_4ce3_43a1_989c_c237e33c1dd8.slice/crio-ecaf9f990fa1a635896a18b1e3af04eadbcff31640fb45ad3c93ca7e43c572d0 WatchSource:0}: Error finding container ecaf9f990fa1a635896a18b1e3af04eadbcff31640fb45ad3c93ca7e43c572d0: Status 404 returned error can't find the container with id ecaf9f990fa1a635896a18b1e3af04eadbcff31640fb45ad3c93ca7e43c572d0
Jul 28 17:19:58.940445 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:58.940417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-stc92"
Jul 28 17:19:59.085757 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:59.085730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-stc92"]
Jul 28 17:19:59.087477 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:19:59.087451    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f367907_cc2a_4cd5_92da_c18390cff656.slice/crio-9d108e681aac84d327c853e9da47615a9cbbfccc52b47b159b3e476cff6a3c45 WatchSource:0}: Error finding container 9d108e681aac84d327c853e9da47615a9cbbfccc52b47b159b3e476cff6a3c45: Status 404 returned error can't find the container with id 9d108e681aac84d327c853e9da47615a9cbbfccc52b47b159b3e476cff6a3c45
Jul 28 17:19:59.642805 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:59.642765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-stc92" event={"ID":"6f367907-cc2a-4cd5-92da-c18390cff656","Type":"ContainerStarted","Data":"9d108e681aac84d327c853e9da47615a9cbbfccc52b47b159b3e476cff6a3c45"}
Jul 28 17:19:59.643839 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:19:59.643812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-74wmk" event={"ID":"7cf64800-4ce3-43a1-989c-c237e33c1dd8","Type":"ContainerStarted","Data":"ecaf9f990fa1a635896a18b1e3af04eadbcff31640fb45ad3c93ca7e43c572d0"}
Jul 28 17:20:01.656258 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:01.656217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-stc92" event={"ID":"6f367907-cc2a-4cd5-92da-c18390cff656","Type":"ContainerStarted","Data":"48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75"}
Jul 28 17:20:01.657713 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:01.657691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-74wmk" event={"ID":"7cf64800-4ce3-43a1-989c-c237e33c1dd8","Type":"ContainerStarted","Data":"32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6"}
Jul 28 17:20:01.672093 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:01.672049    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-stc92" podStartSLOduration=1.654835893 podStartE2EDuration="3.672036172s" totalImagesPullingTime="2.017200279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:01.671481587 +0000 UTC m=+567.906447040" watchObservedRunningTime="2026-07-28 17:20:01.672036172 +0000 UTC m=+567.907001658"
Jul 28 17:20:01.695337 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:01.695291    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-74wmk" podStartSLOduration=1.499671622 podStartE2EDuration="3.695276783s" totalImagesPullingTime="2.195605161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:01.695007087 +0000 UTC m=+567.929972541" watchObservedRunningTime="2026-07-28 17:20:01.695276783 +0000 UTC m=+567.930242257"
Jul 28 17:20:01.727656 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:01.727628    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-74wmk"]
Jul 28 17:20:03.666899 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:03.666857    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-74wmk" podUID="7cf64800-4ce3-43a1-989c-c237e33c1dd8" containerName="authorino" containerID="cri-o://32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6" gracePeriod=30
Jul 28 17:20:03.905375 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:03.905353    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-74wmk"
Jul 28 17:20:04.039772 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.039704    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk4kg\" (UniqueName: \"kubernetes.io/projected/7cf64800-4ce3-43a1-989c-c237e33c1dd8-kube-api-access-kk4kg\") pod \"7cf64800-4ce3-43a1-989c-c237e33c1dd8\" (UID: \"7cf64800-4ce3-43a1-989c-c237e33c1dd8\") "
Jul 28 17:20:04.041841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.041806    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf64800-4ce3-43a1-989c-c237e33c1dd8-kube-api-access-kk4kg" (OuterVolumeSpecName: "kube-api-access-kk4kg") pod "7cf64800-4ce3-43a1-989c-c237e33c1dd8" (UID: "7cf64800-4ce3-43a1-989c-c237e33c1dd8"). InnerVolumeSpecName "kube-api-access-kk4kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:04.140760 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.140734    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk4kg\" (UniqueName: \"kubernetes.io/projected/7cf64800-4ce3-43a1-989c-c237e33c1dd8-kube-api-access-kk4kg\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:20:04.672897 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.672861    2579 generic.go:356] "Generic (PLEG): container finished" podID="7cf64800-4ce3-43a1-989c-c237e33c1dd8" containerID="32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6" exitCode=0
Jul 28 17:20:04.673332 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.672914    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-74wmk"
Jul 28 17:20:04.673332 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.672950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-74wmk" event={"ID":"7cf64800-4ce3-43a1-989c-c237e33c1dd8","Type":"ContainerDied","Data":"32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6"}
Jul 28 17:20:04.673332 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.673004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-74wmk" event={"ID":"7cf64800-4ce3-43a1-989c-c237e33c1dd8","Type":"ContainerDied","Data":"ecaf9f990fa1a635896a18b1e3af04eadbcff31640fb45ad3c93ca7e43c572d0"}
Jul 28 17:20:04.673332 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.673023    2579 scope.go:117] "RemoveContainer" containerID="32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6"
Jul 28 17:20:04.681155 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.681133    2579 scope.go:117] "RemoveContainer" containerID="32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6"
Jul 28 17:20:04.681428 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:20:04.681410    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6\": container with ID starting with 32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6 not found: ID does not exist" containerID="32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6"
Jul 28 17:20:04.681481 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.681437    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6"} err="failed to get container status \"32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6\": rpc error: code = NotFound desc = could not find container \"32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6\": container with ID starting with 32970c3b7a826a11fe7bb08fe723f172877fcd5523e010016b6d89b1d89c32b6 not found: ID does not exist"
Jul 28 17:20:04.689203 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.689177    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-74wmk"]
Jul 28 17:20:04.691320 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:04.691299    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-74wmk"]
Jul 28 17:20:06.284947 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:06.284913    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cf64800-4ce3-43a1-989c-c237e33c1dd8" path="/var/lib/kubelet/pods/7cf64800-4ce3-43a1-989c-c237e33c1dd8/volumes"
Jul 28 17:20:07.599795 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:07.599761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:11.704637 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:11.704602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:17.874478 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:17.874444    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:18.087148 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:18.087112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:20.313242 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:20.313206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:21.544920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.544890    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-ft7f4"]
Jul 28 17:20:21.545254 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.545214    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cf64800-4ce3-43a1-989c-c237e33c1dd8" containerName="authorino"
Jul 28 17:20:21.545254 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.545226    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cf64800-4ce3-43a1-989c-c237e33c1dd8" containerName="authorino"
Jul 28 17:20:21.545340 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.545296    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cf64800-4ce3-43a1-989c-c237e33c1dd8" containerName="authorino"
Jul 28 17:20:21.549669 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.549652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-ft7f4"
Jul 28 17:20:21.552094 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.552074    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 17:20:21.556734 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.556714    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-ft7f4"]
Jul 28 17:20:21.678295 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.678263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4jf\" (UniqueName: \"kubernetes.io/projected/4358a29d-f0d8-48e1-ac74-a6e45464a8c0-kube-api-access-cp4jf\") pod \"authorino-7cf4765b85-ft7f4\" (UID: \"4358a29d-f0d8-48e1-ac74-a6e45464a8c0\") " pod="kuadrant-system/authorino-7cf4765b85-ft7f4"
Jul 28 17:20:21.678469 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.678316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4358a29d-f0d8-48e1-ac74-a6e45464a8c0-tls-cert\") pod \"authorino-7cf4765b85-ft7f4\" (UID: \"4358a29d-f0d8-48e1-ac74-a6e45464a8c0\") " pod="kuadrant-system/authorino-7cf4765b85-ft7f4"
Jul 28 17:20:21.779651 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.779622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4358a29d-f0d8-48e1-ac74-a6e45464a8c0-tls-cert\") pod \"authorino-7cf4765b85-ft7f4\" (UID: \"4358a29d-f0d8-48e1-ac74-a6e45464a8c0\") " pod="kuadrant-system/authorino-7cf4765b85-ft7f4"
Jul 28 17:20:21.779755 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.779701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4jf\" (UniqueName: \"kubernetes.io/projected/4358a29d-f0d8-48e1-ac74-a6e45464a8c0-kube-api-access-cp4jf\") pod \"authorino-7cf4765b85-ft7f4\" (UID: \"4358a29d-f0d8-48e1-ac74-a6e45464a8c0\") " pod="kuadrant-system/authorino-7cf4765b85-ft7f4"
Jul 28 17:20:21.782046 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.782016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4358a29d-f0d8-48e1-ac74-a6e45464a8c0-tls-cert\") pod \"authorino-7cf4765b85-ft7f4\" (UID: \"4358a29d-f0d8-48e1-ac74-a6e45464a8c0\") " pod="kuadrant-system/authorino-7cf4765b85-ft7f4"
Jul 28 17:20:21.789149 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.789128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4jf\" (UniqueName: \"kubernetes.io/projected/4358a29d-f0d8-48e1-ac74-a6e45464a8c0-kube-api-access-cp4jf\") pod \"authorino-7cf4765b85-ft7f4\" (UID: \"4358a29d-f0d8-48e1-ac74-a6e45464a8c0\") " pod="kuadrant-system/authorino-7cf4765b85-ft7f4"
Jul 28 17:20:21.859155 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.859132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-ft7f4"
Jul 28 17:20:21.986516 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:21.986489    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-ft7f4"]
Jul 28 17:20:21.989026 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:20:21.988993    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4358a29d_f0d8_48e1_ac74_a6e45464a8c0.slice/crio-12bf5861d5e69f967cb6bf02f1481180f40513d671ef2c63ae06cedd03020f75 WatchSource:0}: Error finding container 12bf5861d5e69f967cb6bf02f1481180f40513d671ef2c63ae06cedd03020f75: Status 404 returned error can't find the container with id 12bf5861d5e69f967cb6bf02f1481180f40513d671ef2c63ae06cedd03020f75
Jul 28 17:20:22.762540 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:22.762458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-ft7f4" event={"ID":"4358a29d-f0d8-48e1-ac74-a6e45464a8c0","Type":"ContainerStarted","Data":"17206c3f022f5f8a9e08cbe9cc51eff403aee1afa152b3a4148ce51df809ec7c"}
Jul 28 17:20:22.762540 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:22.762501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-ft7f4" event={"ID":"4358a29d-f0d8-48e1-ac74-a6e45464a8c0","Type":"ContainerStarted","Data":"12bf5861d5e69f967cb6bf02f1481180f40513d671ef2c63ae06cedd03020f75"}
Jul 28 17:20:22.782929 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:22.782882    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-ft7f4" podStartSLOduration=1.302204865 podStartE2EDuration="1.782866362s" totalImagesPullingTime="480.661497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:22.777517446 +0000 UTC m=+589.012482899" watchObservedRunningTime="2026-07-28 17:20:22.782866362 +0000 UTC m=+589.017831813"
Jul 28 17:20:22.808661 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:22.808624    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-stc92"]
Jul 28 17:20:22.808802 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:22.808784    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-stc92" podUID="6f367907-cc2a-4cd5-92da-c18390cff656" containerName="authorino" containerID="cri-o://48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75" gracePeriod=30
Jul 28 17:20:23.053843 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.053814    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-stc92"
Jul 28 17:20:23.191670 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.191640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz79d\" (UniqueName: \"kubernetes.io/projected/6f367907-cc2a-4cd5-92da-c18390cff656-kube-api-access-gz79d\") pod \"6f367907-cc2a-4cd5-92da-c18390cff656\" (UID: \"6f367907-cc2a-4cd5-92da-c18390cff656\") "
Jul 28 17:20:23.193660 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.193629    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f367907-cc2a-4cd5-92da-c18390cff656-kube-api-access-gz79d" (OuterVolumeSpecName: "kube-api-access-gz79d") pod "6f367907-cc2a-4cd5-92da-c18390cff656" (UID: "6f367907-cc2a-4cd5-92da-c18390cff656"). InnerVolumeSpecName "kube-api-access-gz79d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:23.292295 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.292242    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz79d\" (UniqueName: \"kubernetes.io/projected/6f367907-cc2a-4cd5-92da-c18390cff656-kube-api-access-gz79d\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:20:23.768574 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.768541    2579 generic.go:356] "Generic (PLEG): container finished" podID="6f367907-cc2a-4cd5-92da-c18390cff656" containerID="48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75" exitCode=0
Jul 28 17:20:23.769019 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.768581    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-stc92"
Jul 28 17:20:23.769019 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.768624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-stc92" event={"ID":"6f367907-cc2a-4cd5-92da-c18390cff656","Type":"ContainerDied","Data":"48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75"}
Jul 28 17:20:23.769019 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.768662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-stc92" event={"ID":"6f367907-cc2a-4cd5-92da-c18390cff656","Type":"ContainerDied","Data":"9d108e681aac84d327c853e9da47615a9cbbfccc52b47b159b3e476cff6a3c45"}
Jul 28 17:20:23.769019 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.768682    2579 scope.go:117] "RemoveContainer" containerID="48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75"
Jul 28 17:20:23.777120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.777104    2579 scope.go:117] "RemoveContainer" containerID="48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75"
Jul 28 17:20:23.777364 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:20:23.777344    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75\": container with ID starting with 48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75 not found: ID does not exist" containerID="48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75"
Jul 28 17:20:23.777474 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.777371    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75"} err="failed to get container status \"48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75\": rpc error: code = NotFound desc = could not find container \"48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75\": container with ID starting with 48875b906786cda459137293a6e64061bd58daeb2264ff4f6d6e30519d31af75 not found: ID does not exist"
Jul 28 17:20:23.789351 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.789330    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-stc92"]
Jul 28 17:20:23.791109 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:23.791089    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-stc92"]
Jul 28 17:20:24.285935 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:24.285901    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f367907-cc2a-4cd5-92da-c18390cff656" path="/var/lib/kubelet/pods/6f367907-cc2a-4cd5-92da-c18390cff656/volumes"
Jul 28 17:20:28.264002 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:28.263963    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:28.342410 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:28.342371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:33.395186 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.395149    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-vp66g"]
Jul 28 17:20:33.395633 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.395338    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g" podUID="ac0ab326-0497-42c0-8def-ab2f26849e13" containerName="manager" containerID="cri-o://711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800" gracePeriod=10
Jul 28 17:20:33.419387 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.419362    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-d9v9x"]
Jul 28 17:20:33.419727 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.419713    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f367907-cc2a-4cd5-92da-c18390cff656" containerName="authorino"
Jul 28 17:20:33.419773 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.419728    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f367907-cc2a-4cd5-92da-c18390cff656" containerName="authorino"
Jul 28 17:20:33.419807 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.419781    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f367907-cc2a-4cd5-92da-c18390cff656" containerName="authorino"
Jul 28 17:20:33.426963 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.426940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:20:33.433523 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.433500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-d9v9x"]
Jul 28 17:20:33.576949 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.576922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78ed5dcf-3cee-4ad7-965f-811fe00995b0-cert\") pod \"kserve-controller-manager-cd4d44c67-d9v9x\" (UID: \"78ed5dcf-3cee-4ad7-965f-811fe00995b0\") " pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:20:33.577065 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.576975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chs8b\" (UniqueName: \"kubernetes.io/projected/78ed5dcf-3cee-4ad7-965f-811fe00995b0-kube-api-access-chs8b\") pod \"kserve-controller-manager-cd4d44c67-d9v9x\" (UID: \"78ed5dcf-3cee-4ad7-965f-811fe00995b0\") " pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:20:33.637264 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.637238    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:20:33.678162 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.678105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78ed5dcf-3cee-4ad7-965f-811fe00995b0-cert\") pod \"kserve-controller-manager-cd4d44c67-d9v9x\" (UID: \"78ed5dcf-3cee-4ad7-965f-811fe00995b0\") " pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:20:33.678261 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.678169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chs8b\" (UniqueName: \"kubernetes.io/projected/78ed5dcf-3cee-4ad7-965f-811fe00995b0-kube-api-access-chs8b\") pod \"kserve-controller-manager-cd4d44c67-d9v9x\" (UID: \"78ed5dcf-3cee-4ad7-965f-811fe00995b0\") " pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:20:33.680290 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.680264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78ed5dcf-3cee-4ad7-965f-811fe00995b0-cert\") pod \"kserve-controller-manager-cd4d44c67-d9v9x\" (UID: \"78ed5dcf-3cee-4ad7-965f-811fe00995b0\") " pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:20:33.686027 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.685995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chs8b\" (UniqueName: \"kubernetes.io/projected/78ed5dcf-3cee-4ad7-965f-811fe00995b0-kube-api-access-chs8b\") pod \"kserve-controller-manager-cd4d44c67-d9v9x\" (UID: \"78ed5dcf-3cee-4ad7-965f-811fe00995b0\") " pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:20:33.767019 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.766994    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:20:33.778974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.778955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrdjj\" (UniqueName: \"kubernetes.io/projected/ac0ab326-0497-42c0-8def-ab2f26849e13-kube-api-access-xrdjj\") pod \"ac0ab326-0497-42c0-8def-ab2f26849e13\" (UID: \"ac0ab326-0497-42c0-8def-ab2f26849e13\") "
Jul 28 17:20:33.779025 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.779016    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac0ab326-0497-42c0-8def-ab2f26849e13-cert\") pod \"ac0ab326-0497-42c0-8def-ab2f26849e13\" (UID: \"ac0ab326-0497-42c0-8def-ab2f26849e13\") "
Jul 28 17:20:33.780991 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.780954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0ab326-0497-42c0-8def-ab2f26849e13-kube-api-access-xrdjj" (OuterVolumeSpecName: "kube-api-access-xrdjj") pod "ac0ab326-0497-42c0-8def-ab2f26849e13" (UID: "ac0ab326-0497-42c0-8def-ab2f26849e13"). InnerVolumeSpecName "kube-api-access-xrdjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:33.780991 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.780985    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0ab326-0497-42c0-8def-ab2f26849e13-cert" (OuterVolumeSpecName: "cert") pod "ac0ab326-0497-42c0-8def-ab2f26849e13" (UID: "ac0ab326-0497-42c0-8def-ab2f26849e13"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:20:33.816843 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.816809    2579 generic.go:356] "Generic (PLEG): container finished" podID="ac0ab326-0497-42c0-8def-ab2f26849e13" containerID="711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800" exitCode=0
Jul 28 17:20:33.816974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.816864    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g"
Jul 28 17:20:33.816974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.816889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g" event={"ID":"ac0ab326-0497-42c0-8def-ab2f26849e13","Type":"ContainerDied","Data":"711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800"}
Jul 28 17:20:33.816974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.816951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-vp66g" event={"ID":"ac0ab326-0497-42c0-8def-ab2f26849e13","Type":"ContainerDied","Data":"4b742ede5f2105f122bbaffd4d256b8d53d66796ed3f277f9b4a01ab9264d964"}
Jul 28 17:20:33.817135 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.816990    2579 scope.go:117] "RemoveContainer" containerID="711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800"
Jul 28 17:20:33.827938 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.827863    2579 scope.go:117] "RemoveContainer" containerID="711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800"
Jul 28 17:20:33.828326 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:20:33.828291    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800\": container with ID starting with 711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800 not found: ID does not exist" containerID="711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800"
Jul 28 17:20:33.828443 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.828340    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800"} err="failed to get container status \"711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800\": rpc error: code = NotFound desc = could not find container \"711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800\": container with ID starting with 711427c150fe5fa26cd3dbde572686d9071a936c8a1015978fd9ee41e82cd800 not found: ID does not exist"
Jul 28 17:20:33.844514 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.844482    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-vp66g"]
Jul 28 17:20:33.847982 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.847957    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-vp66g"]
Jul 28 17:20:33.879953 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.879930    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdjj\" (UniqueName: \"kubernetes.io/projected/ac0ab326-0497-42c0-8def-ab2f26849e13-kube-api-access-xrdjj\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:20:33.879953 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.879952    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac0ab326-0497-42c0-8def-ab2f26849e13-cert\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:20:33.895337 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:33.895317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-d9v9x"]
Jul 28 17:20:33.897592 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:20:33.897569    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ed5dcf_3cee_4ad7_965f_811fe00995b0.slice/crio-bcb1ef1a36507aa4b79840d08440ee54d8f4a923243581faad00d46df467fcc8 WatchSource:0}: Error finding container bcb1ef1a36507aa4b79840d08440ee54d8f4a923243581faad00d46df467fcc8: Status 404 returned error can't find the container with id bcb1ef1a36507aa4b79840d08440ee54d8f4a923243581faad00d46df467fcc8
Jul 28 17:20:34.285669 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:34.285645    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac0ab326-0497-42c0-8def-ab2f26849e13" path="/var/lib/kubelet/pods/ac0ab326-0497-42c0-8def-ab2f26849e13/volumes"
Jul 28 17:20:34.823312 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:34.823265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x" event={"ID":"78ed5dcf-3cee-4ad7-965f-811fe00995b0","Type":"ContainerStarted","Data":"5a262e7fe980a411572bca423d2cc86ba59af22efdd824fc25ffd9096f7bfd95"}
Jul 28 17:20:34.823694 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:34.823339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x" event={"ID":"78ed5dcf-3cee-4ad7-965f-811fe00995b0","Type":"ContainerStarted","Data":"bcb1ef1a36507aa4b79840d08440ee54d8f4a923243581faad00d46df467fcc8"}
Jul 28 17:20:34.823694 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:34.823443    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:20:34.853847 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:34.853801    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x" podStartSLOduration=1.475020271 podStartE2EDuration="1.853783882s" totalImagesPullingTime="378.763611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:34.853449857 +0000 UTC m=+601.088415309" watchObservedRunningTime="2026-07-28 17:20:34.853783882 +0000 UTC m=+601.088749335"
Jul 28 17:20:38.506316 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:38.506282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:38.564154 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:20:38.564120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:05.833205 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:05.833135    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd4d44c67-d9v9x"
Jul 28 17:21:06.859209 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:06.859171    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:08.304884 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:08.304846    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:11.707733 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:11.707690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:11.808476 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:11.808443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:24.124132 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.124093    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qf8qs"]
Jul 28 17:21:24.124628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.124470    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac0ab326-0497-42c0-8def-ab2f26849e13" containerName="manager"
Jul 28 17:21:24.124628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.124482    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac0ab326-0497-42c0-8def-ab2f26849e13" containerName="manager"
Jul 28 17:21:24.124628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.124541    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac0ab326-0497-42c0-8def-ab2f26849e13" containerName="manager"
Jul 28 17:21:24.127705 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.127684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qf8qs"
Jul 28 17:21:24.136441 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.136392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qf8qs"]
Jul 28 17:21:24.265646 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.265610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621a10bc-44cc-4010-97c5-619c9bbd5c3b-model-cache\") pod \"s3-init-qf8qs\" (UID: \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\") " pod="kserve/s3-init-qf8qs"
Jul 28 17:21:24.265805 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.265690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9lk\" (UniqueName: \"kubernetes.io/projected/621a10bc-44cc-4010-97c5-619c9bbd5c3b-kube-api-access-5m9lk\") pod \"s3-init-qf8qs\" (UID: \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\") " pod="kserve/s3-init-qf8qs"
Jul 28 17:21:24.366689 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.366654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9lk\" (UniqueName: \"kubernetes.io/projected/621a10bc-44cc-4010-97c5-619c9bbd5c3b-kube-api-access-5m9lk\") pod \"s3-init-qf8qs\" (UID: \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\") " pod="kserve/s3-init-qf8qs"
Jul 28 17:21:24.366838 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.366735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621a10bc-44cc-4010-97c5-619c9bbd5c3b-model-cache\") pod \"s3-init-qf8qs\" (UID: \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\") " pod="kserve/s3-init-qf8qs"
Jul 28 17:21:24.367082 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.367065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621a10bc-44cc-4010-97c5-619c9bbd5c3b-model-cache\") pod \"s3-init-qf8qs\" (UID: \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\") " pod="kserve/s3-init-qf8qs"
Jul 28 17:21:24.375517 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.375463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9lk\" (UniqueName: \"kubernetes.io/projected/621a10bc-44cc-4010-97c5-619c9bbd5c3b-kube-api-access-5m9lk\") pod \"s3-init-qf8qs\" (UID: \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\") " pod="kserve/s3-init-qf8qs"
Jul 28 17:21:24.438412 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.438373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qf8qs"
Jul 28 17:21:24.567106 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.567081    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qf8qs"]
Jul 28 17:21:24.569208 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:21:24.569181    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod621a10bc_44cc_4010_97c5_619c9bbd5c3b.slice/crio-6e9432eeda497124ad987af25216f2d6033fa1bbc2186b9399b7451cc04ac5ae WatchSource:0}: Error finding container 6e9432eeda497124ad987af25216f2d6033fa1bbc2186b9399b7451cc04ac5ae: Status 404 returned error can't find the container with id 6e9432eeda497124ad987af25216f2d6033fa1bbc2186b9399b7451cc04ac5ae
Jul 28 17:21:24.571823 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:24.571800    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:21:25.060926 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:25.060891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf8qs" event={"ID":"621a10bc-44cc-4010-97c5-619c9bbd5c3b","Type":"ContainerStarted","Data":"6e9432eeda497124ad987af25216f2d6033fa1bbc2186b9399b7451cc04ac5ae"}
Jul 28 17:21:28.077623 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:28.077585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf8qs" event={"ID":"621a10bc-44cc-4010-97c5-619c9bbd5c3b","Type":"ContainerStarted","Data":"b2f9dd557925f1114f5c97fb9c78eb4687bd6ded59fc26d1ee65e7f8e15ff23f"}
Jul 28 17:21:35.111223 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:35.111186    2579 generic.go:356] "Generic (PLEG): container finished" podID="621a10bc-44cc-4010-97c5-619c9bbd5c3b" containerID="b2f9dd557925f1114f5c97fb9c78eb4687bd6ded59fc26d1ee65e7f8e15ff23f" exitCode=0
Jul 28 17:21:35.111715 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:35.111262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf8qs" event={"ID":"621a10bc-44cc-4010-97c5-619c9bbd5c3b","Type":"ContainerDied","Data":"b2f9dd557925f1114f5c97fb9c78eb4687bd6ded59fc26d1ee65e7f8e15ff23f"}
Jul 28 17:21:40.147483 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:40.147449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf8qs" event={"ID":"621a10bc-44cc-4010-97c5-619c9bbd5c3b","Type":"ContainerStarted","Data":"30d98cec25ddcf255028b6a716781ff22887f0e8f147649d8d8b4948363316cb"}
Jul 28 17:21:40.172443 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:21:40.172378    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-qf8qs" podStartSLOduration=2.044537695 podStartE2EDuration="16.172362508s" totalImagesPullingTime="7.411291349s" totalInitContainerRuntime="6.716533464s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:40.170373497 +0000 UTC m=+666.405338940" watchObservedRunningTime="2026-07-28 17:21:40.172362508 +0000 UTC m=+666.407327989"
Jul 28 17:22:44.447621 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:44.447538    2579 generic.go:356] "Generic (PLEG): container finished" podID="621a10bc-44cc-4010-97c5-619c9bbd5c3b" containerID="30d98cec25ddcf255028b6a716781ff22887f0e8f147649d8d8b4948363316cb" exitCode=0
Jul 28 17:22:44.447987 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:44.447611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf8qs" event={"ID":"621a10bc-44cc-4010-97c5-619c9bbd5c3b","Type":"ContainerDied","Data":"30d98cec25ddcf255028b6a716781ff22887f0e8f147649d8d8b4948363316cb"}
Jul 28 17:22:45.582297 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:45.582273    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qf8qs"
Jul 28 17:22:45.636124 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:45.636098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621a10bc-44cc-4010-97c5-619c9bbd5c3b-model-cache\") pod \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\" (UID: \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\") "
Jul 28 17:22:45.636236 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:45.636184    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m9lk\" (UniqueName: \"kubernetes.io/projected/621a10bc-44cc-4010-97c5-619c9bbd5c3b-kube-api-access-5m9lk\") pod \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\" (UID: \"621a10bc-44cc-4010-97c5-619c9bbd5c3b\") "
Jul 28 17:22:45.638144 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:45.638123    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621a10bc-44cc-4010-97c5-619c9bbd5c3b-kube-api-access-5m9lk" (OuterVolumeSpecName: "kube-api-access-5m9lk") pod "621a10bc-44cc-4010-97c5-619c9bbd5c3b" (UID: "621a10bc-44cc-4010-97c5-619c9bbd5c3b"). InnerVolumeSpecName "kube-api-access-5m9lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:22:45.654920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:45.654895    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621a10bc-44cc-4010-97c5-619c9bbd5c3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "621a10bc-44cc-4010-97c5-619c9bbd5c3b" (UID: "621a10bc-44cc-4010-97c5-619c9bbd5c3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:22:45.737460 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:45.737383    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/621a10bc-44cc-4010-97c5-619c9bbd5c3b-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:22:45.737460 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:45.737426    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m9lk\" (UniqueName: \"kubernetes.io/projected/621a10bc-44cc-4010-97c5-619c9bbd5c3b-kube-api-access-5m9lk\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:22:46.458500 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:46.458466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qf8qs" event={"ID":"621a10bc-44cc-4010-97c5-619c9bbd5c3b","Type":"ContainerDied","Data":"6e9432eeda497124ad987af25216f2d6033fa1bbc2186b9399b7451cc04ac5ae"}
Jul 28 17:22:46.458500 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:46.458487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qf8qs"
Jul 28 17:22:46.458500 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:46.458505    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9432eeda497124ad987af25216f2d6033fa1bbc2186b9399b7451cc04ac5ae"
Jul 28 17:22:46.842104 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:46.842064    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:46.943865 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:46.943831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:54.311670 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:54.311635    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:58.959342 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:58.959300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:59.049925 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:22:59.049889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:16.166541 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.166503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:16.518359 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.518288    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"]
Jul 28 17:23:16.518659 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.518646    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="621a10bc-44cc-4010-97c5-619c9bbd5c3b" containerName="download-hf-model"
Jul 28 17:23:16.518711 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.518660    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="621a10bc-44cc-4010-97c5-619c9bbd5c3b" containerName="download-hf-model"
Jul 28 17:23:16.518711 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.518685    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="621a10bc-44cc-4010-97c5-619c9bbd5c3b" containerName="s3-init"
Jul 28 17:23:16.518711 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.518691    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="621a10bc-44cc-4010-97c5-619c9bbd5c3b" containerName="s3-init"
Jul 28 17:23:16.518829 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.518759    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="621a10bc-44cc-4010-97c5-619c9bbd5c3b" containerName="s3-init"
Jul 28 17:23:16.521745 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.521725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:23:16.525173 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.525147    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 28 17:23:16.525268 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.525221    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 28 17:23:16.525324 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.525270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 28 17:23:16.525324 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.525299    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-7n7sc\""
Jul 28 17:23:16.525466 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.525374    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-w452g\""
Jul 28 17:23:16.537083 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.537065    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"]
Jul 28 17:23:16.543635 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.543606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:16.686953 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.686929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb53359-09ff-4f18-89ea-cc289f7b796e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27\" (UID: \"beb53359-09ff-4f18-89ea-cc289f7b796e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:23:16.687072 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.686991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdnd\" (UniqueName: \"kubernetes.io/projected/beb53359-09ff-4f18-89ea-cc289f7b796e-kube-api-access-scdnd\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27\" (UID: \"beb53359-09ff-4f18-89ea-cc289f7b796e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:23:16.787838 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.787774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdnd\" (UniqueName: \"kubernetes.io/projected/beb53359-09ff-4f18-89ea-cc289f7b796e-kube-api-access-scdnd\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27\" (UID: \"beb53359-09ff-4f18-89ea-cc289f7b796e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:23:16.787838 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.787827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb53359-09ff-4f18-89ea-cc289f7b796e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27\" (UID: \"beb53359-09ff-4f18-89ea-cc289f7b796e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:23:16.790247 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.790230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb53359-09ff-4f18-89ea-cc289f7b796e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27\" (UID: \"beb53359-09ff-4f18-89ea-cc289f7b796e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:23:16.795696 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.795679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdnd\" (UniqueName: \"kubernetes.io/projected/beb53359-09ff-4f18-89ea-cc289f7b796e-kube-api-access-scdnd\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27\" (UID: \"beb53359-09ff-4f18-89ea-cc289f7b796e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:23:16.831750 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.831732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:23:16.958306 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:16.958282    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"]
Jul 28 17:23:16.960137 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:23:16.960113    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb53359_09ff_4f18_89ea_cc289f7b796e.slice/crio-93305010fbda4447b2db82e89fe4e561bfabc5c70627f44b25dc8cdcd6b82a39 WatchSource:0}: Error finding container 93305010fbda4447b2db82e89fe4e561bfabc5c70627f44b25dc8cdcd6b82a39: Status 404 returned error can't find the container with id 93305010fbda4447b2db82e89fe4e561bfabc5c70627f44b25dc8cdcd6b82a39
Jul 28 17:23:17.607312 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:17.607277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27" event={"ID":"beb53359-09ff-4f18-89ea-cc289f7b796e","Type":"ContainerStarted","Data":"93305010fbda4447b2db82e89fe4e561bfabc5c70627f44b25dc8cdcd6b82a39"}
Jul 28 17:23:19.620458 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:19.620365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27" event={"ID":"beb53359-09ff-4f18-89ea-cc289f7b796e","Type":"ContainerStarted","Data":"c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3"}
Jul 28 17:23:19.620874 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:19.620582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:23:19.641098 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:19.641049    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27" podStartSLOduration=1.074862371 podStartE2EDuration="3.641036189s" totalImagesPullingTime="2.566173818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:19.64061136 +0000 UTC m=+765.875576815" watchObservedRunningTime="2026-07-28 17:23:19.641036189 +0000 UTC m=+765.876001640"
Jul 28 17:23:26.760299 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.760216    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"]
Jul 28 17:23:26.763652 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.763632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.766165 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.766145    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 28 17:23:26.766307 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.766289    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 28 17:23:26.766307 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.766298    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 28 17:23:26.766457 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.766315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 28 17:23:26.766457 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.766380    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-wkz9n\""
Jul 28 17:23:26.775005 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.774984    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"]
Jul 28 17:23:26.787735 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.787717    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:26.868951 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.868931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.869051 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.868969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.869051 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.868993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.869128 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.869094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.869128 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.869120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e372924-2455-4255-a2c1-1b60e9286813-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.869194 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.869146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ds6\" (UniqueName: \"kubernetes.io/projected/7e372924-2455-4255-a2c1-1b60e9286813-kube-api-access-b4ds6\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.869194 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.869177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970286 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970379 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970449 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970528 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970578 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e372924-2455-4255-a2c1-1b60e9286813-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970635 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ds6\" (UniqueName: \"kubernetes.io/projected/7e372924-2455-4255-a2c1-1b60e9286813-kube-api-access-b4ds6\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970635 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970734 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970734 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970896 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.970980 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.970957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.972541 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.972522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.973083 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.973049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e372924-2455-4255-a2c1-1b60e9286813-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:26.978530 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:26.978508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ds6\" (UniqueName: \"kubernetes.io/projected/7e372924-2455-4255-a2c1-1b60e9286813-kube-api-access-b4ds6\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:27.074880 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:27.074846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:23:27.207784 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:27.207759    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"]
Jul 28 17:23:27.210058 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:23:27.210029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e372924_2455_4255_a2c1_1b60e9286813.slice/crio-dbcea7c07d9ac16ad745486a62c8e34b464ef969cd85533a5e4d97c849a70d40 WatchSource:0}: Error finding container dbcea7c07d9ac16ad745486a62c8e34b464ef969cd85533a5e4d97c849a70d40: Status 404 returned error can't find the container with id dbcea7c07d9ac16ad745486a62c8e34b464ef969cd85533a5e4d97c849a70d40
Jul 28 17:23:27.658890 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:27.658847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" event={"ID":"7e372924-2455-4255-a2c1-1b60e9286813","Type":"ContainerStarted","Data":"a84abcd37c241cbfe450895f5b942427a3604b0ee66487d07a2dd4c4efe97657"}
Jul 28 17:23:27.659075 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:27.658914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" event={"ID":"7e372924-2455-4255-a2c1-1b60e9286813","Type":"ContainerStarted","Data":"dbcea7c07d9ac16ad745486a62c8e34b464ef969cd85533a5e4d97c849a70d40"}
Jul 28 17:23:41.139271 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:41.139209    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:42.740596 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:42.740519    2579 generic.go:356] "Generic (PLEG): container finished" podID="7e372924-2455-4255-a2c1-1b60e9286813" containerID="a84abcd37c241cbfe450895f5b942427a3604b0ee66487d07a2dd4c4efe97657" exitCode=0
Jul 28 17:23:42.740950 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:42.740587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" event={"ID":"7e372924-2455-4255-a2c1-1b60e9286813","Type":"ContainerDied","Data":"a84abcd37c241cbfe450895f5b942427a3604b0ee66487d07a2dd4c4efe97657"}
Jul 28 17:23:50.627148 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:23:50.627115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:24:06.809568 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:06.809324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"]
Jul 28 17:24:06.815089 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:06.815057    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:06.879796 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:06.879744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:10.626600 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:10.626559    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27" podUID="beb53359-09ff-4f18-89ea-cc289f7b796e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:24:12.917194 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:12.917143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" event={"ID":"7e372924-2455-4255-a2c1-1b60e9286813","Type":"ContainerStarted","Data":"ee21955a58fa89ce5baa906d39e08a6572854bfc1524ebff3c6806b1d16fea98"}
Jul 28 17:24:12.942351 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:12.942290    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podStartSLOduration=1.978529757 podStartE2EDuration="46.942273733s" totalImagesPullingTime="29.778549203s" totalInitContainerRuntime="15.185194773s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:12.940774948 +0000 UTC m=+819.175740401" watchObservedRunningTime="2026-07-28 17:24:12.942273733 +0000 UTC m=+819.177239185"
Jul 28 17:24:13.075499 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:13.075464    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:24:17.076039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:17.075994    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:24:17.077847 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:17.077821    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:24:20.626339 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:20.626300    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27" podUID="beb53359-09ff-4f18-89ea-cc289f7b796e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:24:21.835314 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:21.835265    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27" podUID="beb53359-09ff-4f18-89ea-cc289f7b796e" containerName="main" containerID="cri-o://c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3" gracePeriod=45
Jul 28 17:24:22.641984 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.641952    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:24:22.677943 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.677915    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scdnd\" (UniqueName: \"kubernetes.io/projected/beb53359-09ff-4f18-89ea-cc289f7b796e-kube-api-access-scdnd\") pod \"beb53359-09ff-4f18-89ea-cc289f7b796e\" (UID: \"beb53359-09ff-4f18-89ea-cc289f7b796e\") "
Jul 28 17:24:22.678067 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.678011    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb53359-09ff-4f18-89ea-cc289f7b796e-tls-certs\") pod \"beb53359-09ff-4f18-89ea-cc289f7b796e\" (UID: \"beb53359-09ff-4f18-89ea-cc289f7b796e\") "
Jul 28 17:24:22.680040 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.680007    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb53359-09ff-4f18-89ea-cc289f7b796e-kube-api-access-scdnd" (OuterVolumeSpecName: "kube-api-access-scdnd") pod "beb53359-09ff-4f18-89ea-cc289f7b796e" (UID: "beb53359-09ff-4f18-89ea-cc289f7b796e"). InnerVolumeSpecName "kube-api-access-scdnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:24:22.680445 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.680383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb53359-09ff-4f18-89ea-cc289f7b796e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "beb53359-09ff-4f18-89ea-cc289f7b796e" (UID: "beb53359-09ff-4f18-89ea-cc289f7b796e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:22.779557 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.779489    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scdnd\" (UniqueName: \"kubernetes.io/projected/beb53359-09ff-4f18-89ea-cc289f7b796e-kube-api-access-scdnd\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:24:22.779557 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.779515    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb53359-09ff-4f18-89ea-cc289f7b796e-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:24:22.976681 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.976651    2579 generic.go:356] "Generic (PLEG): container finished" podID="beb53359-09ff-4f18-89ea-cc289f7b796e" containerID="c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3" exitCode=0
Jul 28 17:24:22.977066 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.976707    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"
Jul 28 17:24:22.977066 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.976728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27" event={"ID":"beb53359-09ff-4f18-89ea-cc289f7b796e","Type":"ContainerDied","Data":"c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3"}
Jul 28 17:24:22.977066 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.976782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27" event={"ID":"beb53359-09ff-4f18-89ea-cc289f7b796e","Type":"ContainerDied","Data":"93305010fbda4447b2db82e89fe4e561bfabc5c70627f44b25dc8cdcd6b82a39"}
Jul 28 17:24:22.977066 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.976806    2579 scope.go:117] "RemoveContainer" containerID="c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3"
Jul 28 17:24:22.985524 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.985499    2579 scope.go:117] "RemoveContainer" containerID="c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3"
Jul 28 17:24:22.985777 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:24:22.985757    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3\": container with ID starting with c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3 not found: ID does not exist" containerID="c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3"
Jul 28 17:24:22.985849 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:22.985788    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3"} err="failed to get container status \"c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3\": rpc error: code = NotFound desc = could not find container \"c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3\": container with ID starting with c84b6fc0825b242cc128a505b416528af9d1dc14c87d0e785b28be36b26a75d3 not found: ID does not exist"
Jul 28 17:24:23.002365 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:23.002337    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"]
Jul 28 17:24:23.005484 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:23.005463    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-qcw27"]
Jul 28 17:24:24.286110 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:24.286075    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb53359-09ff-4f18-89ea-cc289f7b796e" path="/var/lib/kubelet/pods/beb53359-09ff-4f18-89ea-cc289f7b796e/volumes"
Jul 28 17:24:26.880177 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:26.880138    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:26.976563 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:26.976521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:27.076262 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:27.076216    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:24:32.920314 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:32.920284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:37.075764 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:37.075719    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:24:38.799133 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:38.799089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:38.901819 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:38.901788    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:41.958817 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:41.958780    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:42.205441 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:42.205383    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:47.075746 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:47.075697    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:24:50.481435 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:50.481378    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:57.075873 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:24:57.075830    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:25:00.251048 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:00.251015    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:07.075590 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:07.075551    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:25:17.076199 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:17.076154    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:25:17.128609 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:17.128578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:17.193301 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:17.193269    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:23.069269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:23.069236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:23.165105 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:23.165067    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:27.076235 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:27.076146    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:25:37.075817 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:37.075772    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:25:42.950145 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:42.950102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:43.058266 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:43.058229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:43.234387 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:43.234309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:43.327990 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:43.327952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:47.046231 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:47.046193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:47.076112 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:47.076080    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:25:48.853195 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:48.853153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:52.067246 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:52.067207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:53.903776 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:53.903736    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:54.004425 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:54.004371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:54.105880 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:54.105850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:54.247282 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:54.247247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:54.347953 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:54.347913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:54.846309 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:54.846271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:55.635110 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:55.635069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:55.719675 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:55.719638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:57.076212 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:57.076166    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 28 17:25:57.235188 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:57.235151    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:58.964010 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:25:58.963976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:04.249502 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:04.249457    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:04.997556 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:04.997522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:07.085864 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:07.085832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:26:07.093519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:07.093493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:26:10.510689 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:10.510652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:12.089246 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:12.089208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:14.445872 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:14.445838    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:14.546912 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:14.546884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:14.693661 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:14.693628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:19.434357 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:19.434318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:20.005298 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:20.005259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:20.105681 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:20.105645    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:25.180154 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:25.180117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:25.821679 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:25.821634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:31.854555 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:31.854516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:35.450197 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:35.450152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:35.684538 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:35.684499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:38.047452 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:38.047419    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"]
Jul 28 17:26:38.064731 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:38.064696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:39.986295 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:39.986260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:40.051228 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:40.051191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:45.458233 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:45.458198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:53.070085 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.070045    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main" containerID="cri-o://ee21955a58fa89ce5baa906d39e08a6572854bfc1524ebff3c6806b1d16fea98" gracePeriod=45
Jul 28 17:26:53.691866 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.691836    2579 generic.go:356] "Generic (PLEG): container finished" podID="7e372924-2455-4255-a2c1-1b60e9286813" containerID="ee21955a58fa89ce5baa906d39e08a6572854bfc1524ebff3c6806b1d16fea98" exitCode=0
Jul 28 17:26:53.691974 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.691905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" event={"ID":"7e372924-2455-4255-a2c1-1b60e9286813","Type":"ContainerDied","Data":"ee21955a58fa89ce5baa906d39e08a6572854bfc1524ebff3c6806b1d16fea98"}
Jul 28 17:26:53.824731 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.824708    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:26:53.877246 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.877191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-kserve-provision-location\") pod \"7e372924-2455-4255-a2c1-1b60e9286813\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") "
Jul 28 17:26:53.877360 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.877248    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e372924-2455-4255-a2c1-1b60e9286813-tls-certs\") pod \"7e372924-2455-4255-a2c1-1b60e9286813\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") "
Jul 28 17:26:53.877360 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.877281    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-dshm\") pod \"7e372924-2455-4255-a2c1-1b60e9286813\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") "
Jul 28 17:26:53.877360 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.877304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4ds6\" (UniqueName: \"kubernetes.io/projected/7e372924-2455-4255-a2c1-1b60e9286813-kube-api-access-b4ds6\") pod \"7e372924-2455-4255-a2c1-1b60e9286813\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") "
Jul 28 17:26:53.877360 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.877338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-model-cache\") pod \"7e372924-2455-4255-a2c1-1b60e9286813\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") "
Jul 28 17:26:53.877572 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.877375    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-home\") pod \"7e372924-2455-4255-a2c1-1b60e9286813\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") "
Jul 28 17:26:53.877572 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.877427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-tmp-dir\") pod \"7e372924-2455-4255-a2c1-1b60e9286813\" (UID: \"7e372924-2455-4255-a2c1-1b60e9286813\") "
Jul 28 17:26:53.877695 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.877644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e372924-2455-4255-a2c1-1b60e9286813" (UID: "7e372924-2455-4255-a2c1-1b60e9286813"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:53.878109 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.878085    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-home" (OuterVolumeSpecName: "home") pod "7e372924-2455-4255-a2c1-1b60e9286813" (UID: "7e372924-2455-4255-a2c1-1b60e9286813"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:53.879522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.879498    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-dshm" (OuterVolumeSpecName: "dshm") pod "7e372924-2455-4255-a2c1-1b60e9286813" (UID: "7e372924-2455-4255-a2c1-1b60e9286813"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:53.879823 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.879799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e372924-2455-4255-a2c1-1b60e9286813-kube-api-access-b4ds6" (OuterVolumeSpecName: "kube-api-access-b4ds6") pod "7e372924-2455-4255-a2c1-1b60e9286813" (UID: "7e372924-2455-4255-a2c1-1b60e9286813"). InnerVolumeSpecName "kube-api-access-b4ds6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:53.880077 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.880060    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e372924-2455-4255-a2c1-1b60e9286813-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e372924-2455-4255-a2c1-1b60e9286813" (UID: "7e372924-2455-4255-a2c1-1b60e9286813"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:53.890813 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.890788    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e372924-2455-4255-a2c1-1b60e9286813" (UID: "7e372924-2455-4255-a2c1-1b60e9286813"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:53.932837 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.932814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e372924-2455-4255-a2c1-1b60e9286813" (UID: "7e372924-2455-4255-a2c1-1b60e9286813"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:53.978332 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.978313    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e372924-2455-4255-a2c1-1b60e9286813-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:26:53.978332 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.978333    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:26:53.978470 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.978341    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ds6\" (UniqueName: \"kubernetes.io/projected/7e372924-2455-4255-a2c1-1b60e9286813-kube-api-access-b4ds6\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:26:53.978470 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.978349    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:26:53.978470 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.978357    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:26:53.978470 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.978366    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:26:53.978470 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:53.978373    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e372924-2455-4255-a2c1-1b60e9286813-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:26:54.697599 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:54.697573    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"
Jul 28 17:26:54.697599 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:54.697578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm" event={"ID":"7e372924-2455-4255-a2c1-1b60e9286813","Type":"ContainerDied","Data":"dbcea7c07d9ac16ad745486a62c8e34b464ef969cd85533a5e4d97c849a70d40"}
Jul 28 17:26:54.698078 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:54.697636    2579 scope.go:117] "RemoveContainer" containerID="ee21955a58fa89ce5baa906d39e08a6572854bfc1524ebff3c6806b1d16fea98"
Jul 28 17:26:54.706509 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:54.706488    2579 scope.go:117] "RemoveContainer" containerID="a84abcd37c241cbfe450895f5b942427a3604b0ee66487d07a2dd4c4efe97657"
Jul 28 17:26:54.718635 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:54.718610    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"]
Jul 28 17:26:54.723574 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:54.723555    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8kdjpm"]
Jul 28 17:26:56.284132 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:56.284056    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e372924-2455-4255-a2c1-1b60e9286813" path="/var/lib/kubelet/pods/7e372924-2455-4255-a2c1-1b60e9286813/volumes"
Jul 28 17:26:58.944066 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:58.944021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:26:59.039892 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:26:59.039862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:01.621997 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:01.621961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:05.221250 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:05.221210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:08.576963 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.576918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:08.703112 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.703073    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:08.870834 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.870759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:08.905899 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.905865    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"]
Jul 28 17:27:08.906427 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.906392    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="storage-initializer"
Jul 28 17:27:08.906427 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.906428    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="storage-initializer"
Jul 28 17:27:08.906597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.906442    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main"
Jul 28 17:27:08.906597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.906450    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main"
Jul 28 17:27:08.906597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.906460    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb53359-09ff-4f18-89ea-cc289f7b796e" containerName="main"
Jul 28 17:27:08.906597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.906468    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb53359-09ff-4f18-89ea-cc289f7b796e" containerName="main"
Jul 28 17:27:08.906597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.906544    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb53359-09ff-4f18-89ea-cc289f7b796e" containerName="main"
Jul 28 17:27:08.906597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.906557    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e372924-2455-4255-a2c1-1b60e9286813" containerName="main"
Jul 28 17:27:08.909544 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.909529    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:27:08.909654 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.909637    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:08.912705 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.912688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:08.913920 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.913905    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 17:27:08.914052 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.913947    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 17:27:08.914138 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.914125    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 17:27:08.914291 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.914278    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-qskvq\""
Jul 28 17:27:08.914959 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.914937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-fkhh5\""
Jul 28 17:27:08.915036 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.914971    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 28 17:27:08.915099 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.914971    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-lr6wf\""
Jul 28 17:27:08.922370 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.922352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"]
Jul 28 17:27:08.926827 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.926805    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:27:08.952288 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:08.952257    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:09.000169 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.000145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47442c3d-9a77-40b3-8e3e-7c757041406e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.000258 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.000179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.000258 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.000199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hg2j\" (UniqueName: \"kubernetes.io/projected/47442c3d-9a77-40b3-8e3e-7c757041406e-kube-api-access-7hg2j\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.000258 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.000219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac26624-8588-4126-9e07-4e102ded3818-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc\" (UID: \"7ac26624-8588-4126-9e07-4e102ded3818\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:09.000436 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.000291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.000436 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.000315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.000436 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.000330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6fvr\" (UniqueName: \"kubernetes.io/projected/7ac26624-8588-4126-9e07-4e102ded3818-kube-api-access-d6fvr\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc\" (UID: \"7ac26624-8588-4126-9e07-4e102ded3818\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:09.000436 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.000353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.000436 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.000383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.100781 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.100757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.100883 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.100784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fvr\" (UniqueName: \"kubernetes.io/projected/7ac26624-8588-4126-9e07-4e102ded3818-kube-api-access-d6fvr\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc\" (UID: \"7ac26624-8588-4126-9e07-4e102ded3818\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:09.100883 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.100808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.100883 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.100833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.101237 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.100886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47442c3d-9a77-40b3-8e3e-7c757041406e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.101237 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.100918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.101237 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.100941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hg2j\" (UniqueName: \"kubernetes.io/projected/47442c3d-9a77-40b3-8e3e-7c757041406e-kube-api-access-7hg2j\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.101237 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.100971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac26624-8588-4126-9e07-4e102ded3818-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc\" (UID: \"7ac26624-8588-4126-9e07-4e102ded3818\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:09.101237 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.101028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.101237 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.101220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.101560 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.101497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.101560 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.101504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.101560 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.101552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.103053 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.103032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.103679 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.103659    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47442c3d-9a77-40b3-8e3e-7c757041406e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.103847 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.103826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac26624-8588-4126-9e07-4e102ded3818-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc\" (UID: \"7ac26624-8588-4126-9e07-4e102ded3818\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:09.112047 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.112024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6fvr\" (UniqueName: \"kubernetes.io/projected/7ac26624-8588-4126-9e07-4e102ded3818-kube-api-access-d6fvr\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc\" (UID: \"7ac26624-8588-4126-9e07-4e102ded3818\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:09.115753 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.115734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hg2j\" (UniqueName: \"kubernetes.io/projected/47442c3d-9a77-40b3-8e3e-7c757041406e-kube-api-access-7hg2j\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.221535 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.221454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:09.229609 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.229589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:09.365906 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.365880    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"]
Jul 28 17:27:09.369320 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:27:09.369288    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac26624_8588_4126_9e07_4e102ded3818.slice/crio-78757cdb3613b941b3ab8df9d07f92266167cbe4f851f98c3d9e3d88c5da3c6a WatchSource:0}: Error finding container 78757cdb3613b941b3ab8df9d07f92266167cbe4f851f98c3d9e3d88c5da3c6a: Status 404 returned error can't find the container with id 78757cdb3613b941b3ab8df9d07f92266167cbe4f851f98c3d9e3d88c5da3c6a
Jul 28 17:27:09.395519 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.395497    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:27:09.402355 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:27:09.402328    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47442c3d_9a77_40b3_8e3e_7c757041406e.slice/crio-2082049ae460614da61cecf5e6b1e483643bb9b9bc37e4e8cb787012427651d3 WatchSource:0}: Error finding container 2082049ae460614da61cecf5e6b1e483643bb9b9bc37e4e8cb787012427651d3: Status 404 returned error can't find the container with id 2082049ae460614da61cecf5e6b1e483643bb9b9bc37e4e8cb787012427651d3
Jul 28 17:27:09.770917 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.770879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"47442c3d-9a77-40b3-8e3e-7c757041406e","Type":"ContainerStarted","Data":"bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803"}
Jul 28 17:27:09.771355 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.770942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"47442c3d-9a77-40b3-8e3e-7c757041406e","Type":"ContainerStarted","Data":"2082049ae460614da61cecf5e6b1e483643bb9b9bc37e4e8cb787012427651d3"}
Jul 28 17:27:09.772453 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.772428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc" event={"ID":"7ac26624-8588-4126-9e07-4e102ded3818","Type":"ContainerStarted","Data":"9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa"}
Jul 28 17:27:09.772562 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.772476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc" event={"ID":"7ac26624-8588-4126-9e07-4e102ded3818","Type":"ContainerStarted","Data":"78757cdb3613b941b3ab8df9d07f92266167cbe4f851f98c3d9e3d88c5da3c6a"}
Jul 28 17:27:09.772562 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.772519    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:09.808767 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:09.808720    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc" podStartSLOduration=1.808705206 podStartE2EDuration="1.808705206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:09.807602338 +0000 UTC m=+996.042567783" watchObservedRunningTime="2026-07-28 17:27:09.808705206 +0000 UTC m=+996.043670636"
Jul 28 17:27:11.038672 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:11.038623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:11.107534 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:11.107470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:13.799739 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:13.799705    2579 generic.go:356] "Generic (PLEG): container finished" podID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerID="bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803" exitCode=0
Jul 28 17:27:13.800241 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:13.799782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"47442c3d-9a77-40b3-8e3e-7c757041406e","Type":"ContainerDied","Data":"bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803"}
Jul 28 17:27:14.461107 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:14.461068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:14.806606 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:14.806573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"47442c3d-9a77-40b3-8e3e-7c757041406e","Type":"ContainerStarted","Data":"1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125"}
Jul 28 17:27:14.828382 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:14.828326    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.965210224 podStartE2EDuration="6.828310557s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.863100333s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:14.827117245 +0000 UTC m=+1001.062082713" watchObservedRunningTime="2026-07-28 17:27:14.828310557 +0000 UTC m=+1001.063276044"
Jul 28 17:27:15.230637 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:15.230556    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:19.230556 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:19.230527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:27:19.231703 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:19.231668    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:27:24.302382 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:24.302346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:29.230589 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:29.230548    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:27:39.230721 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:39.230681    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:27:40.778948 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:40.778912    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:27:49.230783 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:49.230734    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:27:59.230471 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:27:59.230433    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:28:09.230342 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:28:09.230300    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:28:19.230975 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:28:19.230929    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:28:29.230540 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:28:29.230447    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:28:35.106107 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:28:35.106058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:39.230543 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:28:39.230506    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:28:49.240000 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:28:49.239967    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:28:49.247652 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:28:49.247625    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:29:29.748901 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:29:29.748851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:10.465058 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:10.464963    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:10.844807 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:10.844775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:16.696500 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:16.696467    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"]
Jul 28 17:30:16.715539 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:16.715508    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:30:16.726443 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:16.726416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:16.788948 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:16.788913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:20.779185 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:20.779148    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc" podUID="7ac26624-8588-4126-9e07-4e102ded3818" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:30:30.443038 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:30.442990    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:30.536917 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:30.536883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:30.778535 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:30.778448    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc" podUID="7ac26624-8588-4126-9e07-4e102ded3818" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:30:31.716371 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:31.716326    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc" podUID="7ac26624-8588-4126-9e07-4e102ded3818" containerName="main" containerID="cri-o://9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa" gracePeriod=45
Jul 28 17:30:31.744811 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:31.744781    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" containerID="cri-o://1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125" gracePeriod=45
Jul 28 17:30:31.970888 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:31.970817    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:30:32.015315 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.015287    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6fvr\" (UniqueName: \"kubernetes.io/projected/7ac26624-8588-4126-9e07-4e102ded3818-kube-api-access-d6fvr\") pod \"7ac26624-8588-4126-9e07-4e102ded3818\" (UID: \"7ac26624-8588-4126-9e07-4e102ded3818\") "
Jul 28 17:30:32.015484 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.015372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac26624-8588-4126-9e07-4e102ded3818-tls-certs\") pod \"7ac26624-8588-4126-9e07-4e102ded3818\" (UID: \"7ac26624-8588-4126-9e07-4e102ded3818\") "
Jul 28 17:30:32.017689 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.017652    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac26624-8588-4126-9e07-4e102ded3818-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ac26624-8588-4126-9e07-4e102ded3818" (UID: "7ac26624-8588-4126-9e07-4e102ded3818"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:32.017689 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.017662    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac26624-8588-4126-9e07-4e102ded3818-kube-api-access-d6fvr" (OuterVolumeSpecName: "kube-api-access-d6fvr") pod "7ac26624-8588-4126-9e07-4e102ded3818" (UID: "7ac26624-8588-4126-9e07-4e102ded3818"). InnerVolumeSpecName "kube-api-access-d6fvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:32.116509 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.116473    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac26624-8588-4126-9e07-4e102ded3818-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.116509 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.116504    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6fvr\" (UniqueName: \"kubernetes.io/projected/7ac26624-8588-4126-9e07-4e102ded3818-kube-api-access-d6fvr\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.241625 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.241540    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 28 17:30:32.492440 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.492358    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:30:32.620675 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.620651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-dshm\") pod \"47442c3d-9a77-40b3-8e3e-7c757041406e\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") "
Jul 28 17:30:32.620826 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.620686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47442c3d-9a77-40b3-8e3e-7c757041406e-tls-certs\") pod \"47442c3d-9a77-40b3-8e3e-7c757041406e\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") "
Jul 28 17:30:32.620826 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.620734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-kserve-provision-location\") pod \"47442c3d-9a77-40b3-8e3e-7c757041406e\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") "
Jul 28 17:30:32.620826 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.620753    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-home\") pod \"47442c3d-9a77-40b3-8e3e-7c757041406e\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") "
Jul 28 17:30:32.620826 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.620796    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-tmp-dir\") pod \"47442c3d-9a77-40b3-8e3e-7c757041406e\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") "
Jul 28 17:30:32.620826 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.620814    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-model-cache\") pod \"47442c3d-9a77-40b3-8e3e-7c757041406e\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") "
Jul 28 17:30:32.621122 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.620842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hg2j\" (UniqueName: \"kubernetes.io/projected/47442c3d-9a77-40b3-8e3e-7c757041406e-kube-api-access-7hg2j\") pod \"47442c3d-9a77-40b3-8e3e-7c757041406e\" (UID: \"47442c3d-9a77-40b3-8e3e-7c757041406e\") "
Jul 28 17:30:32.621203 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.621175    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-model-cache" (OuterVolumeSpecName: "model-cache") pod "47442c3d-9a77-40b3-8e3e-7c757041406e" (UID: "47442c3d-9a77-40b3-8e3e-7c757041406e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:32.621468 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.621446    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-home" (OuterVolumeSpecName: "home") pod "47442c3d-9a77-40b3-8e3e-7c757041406e" (UID: "47442c3d-9a77-40b3-8e3e-7c757041406e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:32.623009 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.622987    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47442c3d-9a77-40b3-8e3e-7c757041406e-kube-api-access-7hg2j" (OuterVolumeSpecName: "kube-api-access-7hg2j") pod "47442c3d-9a77-40b3-8e3e-7c757041406e" (UID: "47442c3d-9a77-40b3-8e3e-7c757041406e"). InnerVolumeSpecName "kube-api-access-7hg2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:32.623218 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.623193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-dshm" (OuterVolumeSpecName: "dshm") pod "47442c3d-9a77-40b3-8e3e-7c757041406e" (UID: "47442c3d-9a77-40b3-8e3e-7c757041406e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:32.623218 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.623205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47442c3d-9a77-40b3-8e3e-7c757041406e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47442c3d-9a77-40b3-8e3e-7c757041406e" (UID: "47442c3d-9a77-40b3-8e3e-7c757041406e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:32.634192 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.634167    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47442c3d-9a77-40b3-8e3e-7c757041406e" (UID: "47442c3d-9a77-40b3-8e3e-7c757041406e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:32.682725 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.682696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47442c3d-9a77-40b3-8e3e-7c757041406e" (UID: "47442c3d-9a77-40b3-8e3e-7c757041406e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:32.721792 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.721764    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.721792 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.721787    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.722126 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.721801    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hg2j\" (UniqueName: \"kubernetes.io/projected/47442c3d-9a77-40b3-8e3e-7c757041406e-kube-api-access-7hg2j\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.722126 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.721812    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.722126 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.721825    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47442c3d-9a77-40b3-8e3e-7c757041406e-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.722126 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.721837    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.722126 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.721851    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47442c3d-9a77-40b3-8e3e-7c757041406e-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.774022 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.773970    2579 generic.go:356] "Generic (PLEG): container finished" podID="7ac26624-8588-4126-9e07-4e102ded3818" containerID="9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa" exitCode=0
Jul 28 17:30:32.774022 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.774013    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"
Jul 28 17:30:32.774178 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.774061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc" event={"ID":"7ac26624-8588-4126-9e07-4e102ded3818","Type":"ContainerDied","Data":"9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa"}
Jul 28 17:30:32.774178 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.774106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc" event={"ID":"7ac26624-8588-4126-9e07-4e102ded3818","Type":"ContainerDied","Data":"78757cdb3613b941b3ab8df9d07f92266167cbe4f851f98c3d9e3d88c5da3c6a"}
Jul 28 17:30:32.774178 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.774134    2579 scope.go:117] "RemoveContainer" containerID="9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa"
Jul 28 17:30:32.775854 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.775833    2579 generic.go:356] "Generic (PLEG): container finished" podID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerID="1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125" exitCode=0
Jul 28 17:30:32.775951 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.775865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"47442c3d-9a77-40b3-8e3e-7c757041406e","Type":"ContainerDied","Data":"1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125"}
Jul 28 17:30:32.775951 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.775899    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:30:32.775951 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.775908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"47442c3d-9a77-40b3-8e3e-7c757041406e","Type":"ContainerDied","Data":"2082049ae460614da61cecf5e6b1e483643bb9b9bc37e4e8cb787012427651d3"}
Jul 28 17:30:32.784844 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.784826    2579 scope.go:117] "RemoveContainer" containerID="9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa"
Jul 28 17:30:32.785101 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:30:32.785083    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa\": container with ID starting with 9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa not found: ID does not exist" containerID="9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa"
Jul 28 17:30:32.785149 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.785110    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa"} err="failed to get container status \"9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa\": rpc error: code = NotFound desc = could not find container \"9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa\": container with ID starting with 9e202cf2c486be1198fc450171ad451c4568c191ebdc314544864651fd9290fa not found: ID does not exist"
Jul 28 17:30:32.785149 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.785132    2579 scope.go:117] "RemoveContainer" containerID="1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125"
Jul 28 17:30:32.792351 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.792330    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"]
Jul 28 17:30:32.792917 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.792906    2579 scope.go:117] "RemoveContainer" containerID="bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803"
Jul 28 17:30:32.796284 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.796264    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew7sqc"]
Jul 28 17:30:32.800644 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.800628    2579 scope.go:117] "RemoveContainer" containerID="1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125"
Jul 28 17:30:32.800879 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:30:32.800861    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125\": container with ID starting with 1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125 not found: ID does not exist" containerID="1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125"
Jul 28 17:30:32.800934 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.800884    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125"} err="failed to get container status \"1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125\": rpc error: code = NotFound desc = could not find container \"1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125\": container with ID starting with 1e092518b9854a813bb3c6797fe22e0bb0ac90d3c7ad6db6f40cc66ce4c49125 not found: ID does not exist"
Jul 28 17:30:32.800934 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.800897    2579 scope.go:117] "RemoveContainer" containerID="bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803"
Jul 28 17:30:32.801167 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:30:32.801146    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803\": container with ID starting with bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803 not found: ID does not exist" containerID="bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803"
Jul 28 17:30:32.801251 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.801175    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803"} err="failed to get container status \"bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803\": rpc error: code = NotFound desc = could not find container \"bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803\": container with ID starting with bbdc51b357fa461e8178e074bdda27d141170efda91c8930c91a18057269a803 not found: ID does not exist"
Jul 28 17:30:32.815283 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.815262    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:30:32.818316 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:32.818297    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:30:34.284908 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:34.284881    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" path="/var/lib/kubelet/pods/47442c3d-9a77-40b3-8e3e-7c757041406e/volumes"
Jul 28 17:30:34.285277 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:34.285262    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac26624-8588-4126-9e07-4e102ded3818" path="/var/lib/kubelet/pods/7ac26624-8588-4126-9e07-4e102ded3818/volumes"
Jul 28 17:30:36.551034 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:36.551000    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:42.084647 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:42.084614    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:42.209336 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:42.209293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:43.693355 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.693321    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"]
Jul 28 17:30:43.695702 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.693680    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac26624-8588-4126-9e07-4e102ded3818" containerName="main"
Jul 28 17:30:43.695702 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.693694    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac26624-8588-4126-9e07-4e102ded3818" containerName="main"
Jul 28 17:30:43.695702 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.693704    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main"
Jul 28 17:30:43.695702 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.693709    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main"
Jul 28 17:30:43.695702 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.693724    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="storage-initializer"
Jul 28 17:30:43.695702 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.693730    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="storage-initializer"
Jul 28 17:30:43.695702 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.693781    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac26624-8588-4126-9e07-4e102ded3818" containerName="main"
Jul 28 17:30:43.695702 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.693792    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="47442c3d-9a77-40b3-8e3e-7c757041406e" containerName="main"
Jul 28 17:30:43.696769 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.696753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.699687 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.699662    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 28 17:30:43.699806 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.699683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-5svxx\""
Jul 28 17:30:43.699806 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.699671    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 28 17:30:43.699806 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.699709    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 28 17:30:43.699806 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.699772    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 28 17:30:43.710790 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.710767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"]
Jul 28 17:30:43.722094 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.722065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:43.805239 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.805212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-home\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.805239 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.805239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.805387 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.805264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8797f78-2333-45d2-aabe-755fd05d755e-tls-certs\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.805387 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.805311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-tmp-dir\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.805387 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.805348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-dshm\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.805556 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.805385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrj7v\" (UniqueName: \"kubernetes.io/projected/d8797f78-2333-45d2-aabe-755fd05d755e-kube-api-access-qrj7v\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.805556 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.805442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-model-cache\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906300 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-home\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8797f78-2333-45d2-aabe-755fd05d755e-tls-certs\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-tmp-dir\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-dshm\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrj7v\" (UniqueName: \"kubernetes.io/projected/d8797f78-2333-45d2-aabe-755fd05d755e-kube-api-access-qrj7v\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-model-cache\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906772 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-home\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906834 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-tmp-dir\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906834 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.906906 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.906865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-model-cache\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.908724 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.908705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8797f78-2333-45d2-aabe-755fd05d755e-dshm\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.909017 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.908997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8797f78-2333-45d2-aabe-755fd05d755e-tls-certs\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:43.914848 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:43.914825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrj7v\" (UniqueName: \"kubernetes.io/projected/d8797f78-2333-45d2-aabe-755fd05d755e-kube-api-access-qrj7v\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-7dpxq\" (UID: \"d8797f78-2333-45d2-aabe-755fd05d755e\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:44.007161 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:44.007098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"
Jul 28 17:30:44.015595 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:44.015572    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:44.139786 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:44.139756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq"]
Jul 28 17:30:44.142391 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:30:44.142367    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8797f78_2333_45d2_aabe_755fd05d755e.slice/crio-9eb17d7173607357b91980e70449840aae44c45bc6f9af094a177074e9d04bff WatchSource:0}: Error finding container 9eb17d7173607357b91980e70449840aae44c45bc6f9af094a177074e9d04bff: Status 404 returned error can't find the container with id 9eb17d7173607357b91980e70449840aae44c45bc6f9af094a177074e9d04bff
Jul 28 17:30:44.834777 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:44.834739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq" event={"ID":"d8797f78-2333-45d2-aabe-755fd05d755e","Type":"ContainerStarted","Data":"7fc8850720f50f1b56650fb3f28676624d3806a090801e459c07b28775585c3a"}
Jul 28 17:30:44.835249 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:44.834793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-7dpxq" event={"ID":"d8797f78-2333-45d2-aabe-755fd05d755e","Type":"ContainerStarted","Data":"9eb17d7173607357b91980e70449840aae44c45bc6f9af094a177074e9d04bff"}
Jul 28 17:30:51.790112 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:51.790062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:51.886229 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:51.886198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:57.987445 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:30:57.987389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:03.627134 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:03.627093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:03.729985 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:03.729950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:03.982440 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:03.982344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:14.779430 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:14.779360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:14.985718 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:14.985685    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"]
Jul 28 17:31:14.989215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:14.989191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:14.991824 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:14.991780    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-lfd5h\""
Jul 28 17:31:14.993033 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:14.992817    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-vjk54\""
Jul 28 17:31:14.993033 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:14.992893    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 28 17:31:14.993033 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:14.992985    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 28 17:31:14.993257 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:14.993109    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 28 17:31:15.003479 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.003453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"]
Jul 28 17:31:15.055881 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.055854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t878\" (UniqueName: \"kubernetes.io/projected/2156070b-680b-4fca-84eb-b09a6c84a43a-kube-api-access-5t878\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42\" (UID: \"2156070b-680b-4fca-84eb-b09a6c84a43a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:15.056017 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.055898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2156070b-680b-4fca-84eb-b09a6c84a43a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42\" (UID: \"2156070b-680b-4fca-84eb-b09a6c84a43a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:15.156563 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.156531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t878\" (UniqueName: \"kubernetes.io/projected/2156070b-680b-4fca-84eb-b09a6c84a43a-kube-api-access-5t878\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42\" (UID: \"2156070b-680b-4fca-84eb-b09a6c84a43a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:15.156698 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.156579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2156070b-680b-4fca-84eb-b09a6c84a43a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42\" (UID: \"2156070b-680b-4fca-84eb-b09a6c84a43a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:15.159224 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.159192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2156070b-680b-4fca-84eb-b09a6c84a43a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42\" (UID: \"2156070b-680b-4fca-84eb-b09a6c84a43a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:15.164902 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.164882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t878\" (UniqueName: \"kubernetes.io/projected/2156070b-680b-4fca-84eb-b09a6c84a43a-kube-api-access-5t878\") pod \"scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42\" (UID: \"2156070b-680b-4fca-84eb-b09a6c84a43a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:15.302175 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.302152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:15.435585 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.435478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"]
Jul 28 17:31:15.437968 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:31:15.437931    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2156070b_680b_4fca_84eb_b09a6c84a43a.slice/crio-90148facc296d744c530aa6b7d0e4785334c1d4ff739ca2e683ba2053705b42d WatchSource:0}: Error finding container 90148facc296d744c530aa6b7d0e4785334c1d4ff739ca2e683ba2053705b42d: Status 404 returned error can't find the container with id 90148facc296d744c530aa6b7d0e4785334c1d4ff739ca2e683ba2053705b42d
Jul 28 17:31:15.979680 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.979643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42" event={"ID":"2156070b-680b-4fca-84eb-b09a6c84a43a","Type":"ContainerStarted","Data":"108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b"}
Jul 28 17:31:15.980059 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.979704    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:15.980059 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:15.979720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42" event={"ID":"2156070b-680b-4fca-84eb-b09a6c84a43a","Type":"ContainerStarted","Data":"90148facc296d744c530aa6b7d0e4785334c1d4ff739ca2e683ba2053705b42d"}
Jul 28 17:31:16.000793 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:16.000727    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42" podStartSLOduration=2.000690194 podStartE2EDuration="2.000690194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:15.997291569 +0000 UTC m=+1242.232257015" watchObservedRunningTime="2026-07-28 17:31:16.000690194 +0000 UTC m=+1242.235655648"
Jul 28 17:31:16.839632 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:16.839596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:46.986819 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:46.986789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:31:53.804760 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:53.804725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:56.118039 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:56.118000    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:56.124229 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:56.124203    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"]
Jul 28 17:31:56.201748 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:56.201711    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:56.986244 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:31:56.986208    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42" podUID="2156070b-680b-4fca-84eb-b09a6c84a43a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:32:06.986393 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:06.986352    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42" podUID="2156070b-680b-4fca-84eb-b09a6c84a43a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:32:11.144558 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:11.144518    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42" podUID="2156070b-680b-4fca-84eb-b09a6c84a43a" containerName="main" containerID="cri-o://108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b" gracePeriod=45
Jul 28 17:32:11.385888 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:11.385866    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:32:11.506334 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:11.506263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t878\" (UniqueName: \"kubernetes.io/projected/2156070b-680b-4fca-84eb-b09a6c84a43a-kube-api-access-5t878\") pod \"2156070b-680b-4fca-84eb-b09a6c84a43a\" (UID: \"2156070b-680b-4fca-84eb-b09a6c84a43a\") "
Jul 28 17:32:11.506334 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:11.506297    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2156070b-680b-4fca-84eb-b09a6c84a43a-tls-certs\") pod \"2156070b-680b-4fca-84eb-b09a6c84a43a\" (UID: \"2156070b-680b-4fca-84eb-b09a6c84a43a\") "
Jul 28 17:32:11.508293 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:11.508267    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2156070b-680b-4fca-84eb-b09a6c84a43a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2156070b-680b-4fca-84eb-b09a6c84a43a" (UID: "2156070b-680b-4fca-84eb-b09a6c84a43a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:11.508413 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:11.508307    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2156070b-680b-4fca-84eb-b09a6c84a43a-kube-api-access-5t878" (OuterVolumeSpecName: "kube-api-access-5t878") pod "2156070b-680b-4fca-84eb-b09a6c84a43a" (UID: "2156070b-680b-4fca-84eb-b09a6c84a43a"). InnerVolumeSpecName "kube-api-access-5t878". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:11.607336 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:11.607310    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t878\" (UniqueName: \"kubernetes.io/projected/2156070b-680b-4fca-84eb-b09a6c84a43a-kube-api-access-5t878\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:32:11.607336 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:11.607333    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2156070b-680b-4fca-84eb-b09a6c84a43a-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:32:12.241886 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.241849    2579 generic.go:356] "Generic (PLEG): container finished" podID="2156070b-680b-4fca-84eb-b09a6c84a43a" containerID="108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b" exitCode=0
Jul 28 17:32:12.242256 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.241908    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"
Jul 28 17:32:12.242256 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.241913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42" event={"ID":"2156070b-680b-4fca-84eb-b09a6c84a43a","Type":"ContainerDied","Data":"108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b"}
Jul 28 17:32:12.242256 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.242052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42" event={"ID":"2156070b-680b-4fca-84eb-b09a6c84a43a","Type":"ContainerDied","Data":"90148facc296d744c530aa6b7d0e4785334c1d4ff739ca2e683ba2053705b42d"}
Jul 28 17:32:12.242256 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.242083    2579 scope.go:117] "RemoveContainer" containerID="108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b"
Jul 28 17:32:12.250874 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.250857    2579 scope.go:117] "RemoveContainer" containerID="108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b"
Jul 28 17:32:12.251127 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:32:12.251109    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b\": container with ID starting with 108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b not found: ID does not exist" containerID="108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b"
Jul 28 17:32:12.251177 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.251134    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b"} err="failed to get container status \"108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b\": rpc error: code = NotFound desc = could not find container \"108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b\": container with ID starting with 108122bd7d17d81c565c085dca5bf5190a793f1314120cb50c310877e22ec22b not found: ID does not exist"
Jul 28 17:32:12.262707 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.262669    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"]
Jul 28 17:32:12.268642 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.268622    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-6bbd5gkg42"]
Jul 28 17:32:12.284409 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:12.284375    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2156070b-680b-4fca-84eb-b09a6c84a43a" path="/var/lib/kubelet/pods/2156070b-680b-4fca-84eb-b09a6c84a43a/volumes"
Jul 28 17:32:15.783575 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:15.783544    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:15.879444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:15.879395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:18.086675 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:18.086640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:18.323961 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:18.323933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:18.548090 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:18.548062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:22.165340 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:22.165295    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:27.732896 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:27.732860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:27.832305 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:27.832271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:31.274153 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:31.274093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:31.328557 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:31.328518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:33.759507 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:33.759477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:51.715522 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:51.715466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:51.784961 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:32:51.784919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:11.765850 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:11.765768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:11.870225 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:11.870193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:18.024676 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:18.024641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:21.192674 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.192632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:21.324198 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.324161    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"]
Jul 28 17:33:21.324704 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.324688    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2156070b-680b-4fca-84eb-b09a6c84a43a" containerName="main"
Jul 28 17:33:21.324753 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.324708    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2156070b-680b-4fca-84eb-b09a6c84a43a" containerName="main"
Jul 28 17:33:21.324814 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.324804    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2156070b-680b-4fca-84eb-b09a6c84a43a" containerName="main"
Jul 28 17:33:21.328104 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.328083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:21.330728 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.330707    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 28 17:33:21.331583 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.331557    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 28 17:33:21.331807 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.331784    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 28 17:33:21.331885 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.331814    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-bdwth\""
Jul 28 17:33:21.331885 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.331839    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-hdwrw\""
Jul 28 17:33:21.337409 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.337375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"]
Jul 28 17:33:21.355632 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.355595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:21.390119 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.390082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxz4v\" (UniqueName: \"kubernetes.io/projected/9c4b7c68-509e-4963-9c15-b5f4fce22d92-kube-api-access-zxz4v\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576\" (UID: \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:21.390287 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.390153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4b7c68-509e-4963-9c15-b5f4fce22d92-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576\" (UID: \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:21.490739 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.490644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxz4v\" (UniqueName: \"kubernetes.io/projected/9c4b7c68-509e-4963-9c15-b5f4fce22d92-kube-api-access-zxz4v\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576\" (UID: \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:21.490739 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.490700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4b7c68-509e-4963-9c15-b5f4fce22d92-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576\" (UID: \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:21.493136 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.493113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4b7c68-509e-4963-9c15-b5f4fce22d92-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576\" (UID: \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:21.500206 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.500174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxz4v\" (UniqueName: \"kubernetes.io/projected/9c4b7c68-509e-4963-9c15-b5f4fce22d92-kube-api-access-zxz4v\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576\" (UID: \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:21.640074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.640032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:21.775612 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:21.775586    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"]
Jul 28 17:33:21.777589 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:33:21.777559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4b7c68_509e_4963_9c15_b5f4fce22d92.slice/crio-57349484e3f844eadb4a312ea367490e8ff515bac3eadbe2066eefbb1bf6bc5a WatchSource:0}: Error finding container 57349484e3f844eadb4a312ea367490e8ff515bac3eadbe2066eefbb1bf6bc5a: Status 404 returned error can't find the container with id 57349484e3f844eadb4a312ea367490e8ff515bac3eadbe2066eefbb1bf6bc5a
Jul 28 17:33:22.559037 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:22.559002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576" event={"ID":"9c4b7c68-509e-4963-9c15-b5f4fce22d92","Type":"ContainerStarted","Data":"fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7"}
Jul 28 17:33:22.559555 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:22.559063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576" event={"ID":"9c4b7c68-509e-4963-9c15-b5f4fce22d92","Type":"ContainerStarted","Data":"57349484e3f844eadb4a312ea367490e8ff515bac3eadbe2066eefbb1bf6bc5a"}
Jul 28 17:33:22.559555 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:22.559094    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:22.578699 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:22.578640    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576" podStartSLOduration=1.578620439 podStartE2EDuration="1.578620439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:22.578002579 +0000 UTC m=+1368.812968030" watchObservedRunningTime="2026-07-28 17:33:22.578620439 +0000 UTC m=+1368.813585891"
Jul 28 17:33:23.794532 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:23.794496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:23.897534 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:23.897499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:28.278766 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:28.278726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:53.565111 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:53.565077    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:33:54.907753 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:54.907721    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"]
Jul 28 17:33:54.910979 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:54.910951    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:55.007686 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:33:55.007652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:03.565424 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:03.565368    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576" podUID="9c4b7c68-509e-4963-9c15-b5f4fce22d92" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:34:09.928380 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:09.928333    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576" podUID="9c4b7c68-509e-4963-9c15-b5f4fce22d92" containerName="main" containerID="cri-o://fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7" gracePeriod=45
Jul 28 17:34:10.173188 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.173164    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:34:10.182634 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.182594    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4b7c68-509e-4963-9c15-b5f4fce22d92-tls-certs\") pod \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\" (UID: \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\") "
Jul 28 17:34:10.182719 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.182643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxz4v\" (UniqueName: \"kubernetes.io/projected/9c4b7c68-509e-4963-9c15-b5f4fce22d92-kube-api-access-zxz4v\") pod \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\" (UID: \"9c4b7c68-509e-4963-9c15-b5f4fce22d92\") "
Jul 28 17:34:10.184666 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.184641    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4b7c68-509e-4963-9c15-b5f4fce22d92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c4b7c68-509e-4963-9c15-b5f4fce22d92" (UID: "9c4b7c68-509e-4963-9c15-b5f4fce22d92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:34:10.184666 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.184654    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4b7c68-509e-4963-9c15-b5f4fce22d92-kube-api-access-zxz4v" (OuterVolumeSpecName: "kube-api-access-zxz4v") pod "9c4b7c68-509e-4963-9c15-b5f4fce22d92" (UID: "9c4b7c68-509e-4963-9c15-b5f4fce22d92"). InnerVolumeSpecName "kube-api-access-zxz4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:34:10.283597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.283573    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4b7c68-509e-4963-9c15-b5f4fce22d92-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:34:10.283597 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.283595    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxz4v\" (UniqueName: \"kubernetes.io/projected/9c4b7c68-509e-4963-9c15-b5f4fce22d92-kube-api-access-zxz4v\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:34:10.786823 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.786780    2579 generic.go:356] "Generic (PLEG): container finished" podID="9c4b7c68-509e-4963-9c15-b5f4fce22d92" containerID="fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7" exitCode=0
Jul 28 17:34:10.787005 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.786830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576" event={"ID":"9c4b7c68-509e-4963-9c15-b5f4fce22d92","Type":"ContainerDied","Data":"fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7"}
Jul 28 17:34:10.787005 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.786841    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"
Jul 28 17:34:10.787005 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.786912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576" event={"ID":"9c4b7c68-509e-4963-9c15-b5f4fce22d92","Type":"ContainerDied","Data":"57349484e3f844eadb4a312ea367490e8ff515bac3eadbe2066eefbb1bf6bc5a"}
Jul 28 17:34:10.787005 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.786945    2579 scope.go:117] "RemoveContainer" containerID="fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7"
Jul 28 17:34:10.795701 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.795683    2579 scope.go:117] "RemoveContainer" containerID="fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7"
Jul 28 17:34:10.795941 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:34:10.795923    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7\": container with ID starting with fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7 not found: ID does not exist" containerID="fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7"
Jul 28 17:34:10.795987 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.795948    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7"} err="failed to get container status \"fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7\": rpc error: code = NotFound desc = could not find container \"fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7\": container with ID starting with fbeb65184504403894154c9731cfd3b2ccc8d0e6ffbac2264d8166cf9763a5f7 not found: ID does not exist"
Jul 28 17:34:10.804057 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.804035    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"]
Jul 28 17:34:10.807392 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:10.807368    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587s4576"]
Jul 28 17:34:12.285349 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:12.285315    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c4b7c68-509e-4963-9c15-b5f4fce22d92" path="/var/lib/kubelet/pods/9c4b7c68-509e-4963-9c15-b5f4fce22d92/volumes"
Jul 28 17:34:14.606713 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:14.606685    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:14.705484 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:14.705454    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:20.868180 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:20.868143    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:22.280220 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:22.280180    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:22.531786 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:22.531698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:26.473055 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:26.472974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:26.544644 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:26.544607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:28.976014 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:28.975976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:42.931495 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:42.931459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:53.457363 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:53.457322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:53.517879 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:53.517840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:55.767428 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:55.767354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:55.835038 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:34:55.835005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:15.794395 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:35:15.794354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:15.885238 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:35:15.885211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:21.819955 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:35:21.819892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:28.161530 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:35:28.161487    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:28.217859 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:35:28.217824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:38.575986 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:35:38.575941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:38.887744 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:35:38.887641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:47.464765 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:35:47.464729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:12.262961 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:12.262852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:12.362652 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:12.362597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:32.091812 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:32.091775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:33.299602 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:33.299569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:38.056671 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:38.056631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:42.135693 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.135657    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"]
Jul 28 17:36:42.136045 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.136015    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4b7c68-509e-4963-9c15-b5f4fce22d92" containerName="main"
Jul 28 17:36:42.136045 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.136026    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4b7c68-509e-4963-9c15-b5f4fce22d92" containerName="main"
Jul 28 17:36:42.136123 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.136090    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c4b7c68-509e-4963-9c15-b5f4fce22d92" containerName="main"
Jul 28 17:36:42.139002 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.138985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.142361 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.142338    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 17:36:42.142509 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.142342    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 17:36:42.142509 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.142346    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 17:36:42.142509 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.142413    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-74x8n\""
Jul 28 17:36:42.150111 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.150088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"]
Jul 28 17:36:42.165690 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.165658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:42.207934 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.207907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.208038 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.207949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pbrs\" (UniqueName: \"kubernetes.io/projected/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-kube-api-access-5pbrs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.208080 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.208040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.208080 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.208064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.208158 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.208083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.208158 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.208112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.308935 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.308903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.309098 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.308943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.309098 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.308970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.309098 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.309007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.309098 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.309056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.309098 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.309097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pbrs\" (UniqueName: \"kubernetes.io/projected/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-kube-api-access-5pbrs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.309442 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.309388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.309568 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.309503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.309568 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.309504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.311227 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.311208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.311491 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.311475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.320393 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.320363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pbrs\" (UniqueName: \"kubernetes.io/projected/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-kube-api-access-5pbrs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.374439 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.374373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:42.450113 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.450047    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:42.584936 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.584909    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"]
Jul 28 17:36:42.586691 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:36:42.586660    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d3e927_f8c5_4209_a5af_9c85dbcdf1c9.slice/crio-7983e45eefb5e84dec5a27a653280eff2a34379f43e517c45bf8ac940a47e745 WatchSource:0}: Error finding container 7983e45eefb5e84dec5a27a653280eff2a34379f43e517c45bf8ac940a47e745: Status 404 returned error can't find the container with id 7983e45eefb5e84dec5a27a653280eff2a34379f43e517c45bf8ac940a47e745
Jul 28 17:36:42.588623 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:42.588607    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:36:43.488112 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:43.488067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2" event={"ID":"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9","Type":"ContainerStarted","Data":"7983e45eefb5e84dec5a27a653280eff2a34379f43e517c45bf8ac940a47e745"}
Jul 28 17:36:44.499746 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:44.499652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2" event={"ID":"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9","Type":"ContainerStarted","Data":"514a2006fbd5918dce596e604dc7da9e8f81e79691499df779edda2266a9ec8a"}
Jul 28 17:36:44.518497 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:44.518447    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2" podStartSLOduration=0.85699104 podStartE2EDuration="2.518431687s" totalImagesPullingTime="1.661440647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:44.516595246 +0000 UTC m=+1570.751560704" watchObservedRunningTime="2026-07-28 17:36:44.518431687 +0000 UTC m=+1570.753397137"
Jul 28 17:36:44.616885 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:44.616854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:44.717776 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:44.717746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:36:45.450530 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:45.450498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:52.450348 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:52.450314    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:52.462425 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:52.462387    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:36:52.547684 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:36:52.547661    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:37:00.138757 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:37:00.138714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:19.892074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:19.892036    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"]
Jul 28 17:39:19.902701 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:39:19.902658    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:39:19.902701 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:39:19.902701 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:39:19.902701 ip-10-0-137-238 kubenswrapper[2579]:  > containerID="514a2006fbd5918dce596e604dc7da9e8f81e79691499df779edda2266a9ec8a" cmd=["/bin/sleep","15"]
Jul 28 17:39:19.902985 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:19.902732    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2" message="" err=<
Jul 28 17:39:19.902985 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:39:19.902985 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:39:19.902985 ip-10-0-137-238 kubenswrapper[2579]:  >
Jul 28 17:39:19.902985 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:39:19.902748    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:39:19.902985 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:39:19.902985 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:39:19.902985 ip-10-0-137-238 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2" podUID="34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" containerName="main" containerID="cri-o://514a2006fbd5918dce596e604dc7da9e8f81e79691499df779edda2266a9ec8a"
Jul 28 17:39:19.902985 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:19.902798    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2" podUID="34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" containerName="main" containerID="cri-o://514a2006fbd5918dce596e604dc7da9e8f81e79691499df779edda2266a9ec8a" gracePeriod=60
Jul 28 17:39:19.933065 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:19.933035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:20.013283 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.013257    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:20.217849 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.217820    2579 generic.go:356] "Generic (PLEG): container finished" podID="34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" containerID="514a2006fbd5918dce596e604dc7da9e8f81e79691499df779edda2266a9ec8a" exitCode=0
Jul 28 17:39:20.217990 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.217899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2" event={"ID":"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9","Type":"ContainerDied","Data":"514a2006fbd5918dce596e604dc7da9e8f81e79691499df779edda2266a9ec8a"}
Jul 28 17:39:20.273211 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.273178    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:39:20.352136 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352100    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tls-certs\") pod \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") "
Jul 28 17:39:20.352302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352157    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tmp-dir\") pod \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") "
Jul 28 17:39:20.352302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-model-cache\") pod \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") "
Jul 28 17:39:20.352302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pbrs\" (UniqueName: \"kubernetes.io/projected/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-kube-api-access-5pbrs\") pod \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") "
Jul 28 17:39:20.352302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-home\") pod \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") "
Jul 28 17:39:20.352302 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-dshm\") pod \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\" (UID: \"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9\") "
Jul 28 17:39:20.352588 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352422    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" (UID: "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.352588 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352529    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-home" (OuterVolumeSpecName: "home") pod "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" (UID: "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.352688 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352586    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.352783 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.352758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" (UID: "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.354718 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.354693    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" (UID: "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:39:20.354982 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.354954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-kube-api-access-5pbrs" (OuterVolumeSpecName: "kube-api-access-5pbrs") pod "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" (UID: "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9"). InnerVolumeSpecName "kube-api-access-5pbrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:39:20.360524 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.357496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-dshm" (OuterVolumeSpecName: "dshm") pod "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" (UID: "34d3e927-f8c5-4209-a5af-9c85dbcdf1c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.453444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.453326    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.453444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.453353    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.453444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.453365    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pbrs\" (UniqueName: \"kubernetes.io/projected/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-kube-api-access-5pbrs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.453444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.453374    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.453444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:20.453382    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:39:21.224975 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:21.224947    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"
Jul 28 17:39:21.224975 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:21.224960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2" event={"ID":"34d3e927-f8c5-4209-a5af-9c85dbcdf1c9","Type":"ContainerDied","Data":"7983e45eefb5e84dec5a27a653280eff2a34379f43e517c45bf8ac940a47e745"}
Jul 28 17:39:21.225503 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:21.225017    2579 scope.go:117] "RemoveContainer" containerID="514a2006fbd5918dce596e604dc7da9e8f81e79691499df779edda2266a9ec8a"
Jul 28 17:39:21.253109 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:21.253081    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"]
Jul 28 17:39:21.256174 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:21.256148    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-794df6ffb4-h8tg2"]
Jul 28 17:39:22.285014 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:22.284977    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" path="/var/lib/kubelet/pods/34d3e927-f8c5-4209-a5af-9c85dbcdf1c9/volumes"
Jul 28 17:39:39.778176 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:39.778139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:39.923143 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:39.923110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:45.718694 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:45.718662    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:47.264048 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:47.264014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:47.464692 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:47.464657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:51.804733 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:51.804698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:51.869830 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:51.869796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:52.356793 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:52.356762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:39:52.459090 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:39:52.459060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:04.775813 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:04.775775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:21.510202 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:21.510158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:21.580020 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:21.579991    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:41.286001 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:41.285965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:41.380723 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:41.380690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:45.676097 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.676062    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"]
Jul 28 17:40:45.678470 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.676438    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" containerName="main"
Jul 28 17:40:45.678470 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.676459    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" containerName="main"
Jul 28 17:40:45.678470 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.676558    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d3e927-f8c5-4209-a5af-9c85dbcdf1c9" containerName="main"
Jul 28 17:40:45.679528 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.679503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.681776 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.681752    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 28 17:40:45.682057 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.682024    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 17:40:45.682772 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.682754    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 17:40:45.682825 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.682776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-h6ncf\""
Jul 28 17:40:45.682825 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.682754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 17:40:45.695020 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.694997    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"]
Jul 28 17:40:45.698434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.698412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:45.757657 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.757618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.757657 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.757661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.757881 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.757689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.757881 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.757741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0857cc69-854f-4f1c-a8de-33eb3b87062d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.757881 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.757786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.757881 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.757835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfsjr\" (UniqueName: \"kubernetes.io/projected/0857cc69-854f-4f1c-a8de-33eb3b87062d-kube-api-access-pfsjr\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.758046 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.757891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859167 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859352 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859352 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859352 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859352 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0857cc69-854f-4f1c-a8de-33eb3b87062d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859352 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859654 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfsjr\" (UniqueName: \"kubernetes.io/projected/0857cc69-854f-4f1c-a8de-33eb3b87062d-kube-api-access-pfsjr\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859654 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859757 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859757 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.859838 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.859778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.861440 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.861424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.861709 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.861691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0857cc69-854f-4f1c-a8de-33eb3b87062d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.870863 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.870834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfsjr\" (UniqueName: \"kubernetes.io/projected/0857cc69-854f-4f1c-a8de-33eb3b87062d-kube-api-access-pfsjr\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-r47pn\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:45.937790 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.937716    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:45.989603 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:45.989577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:46.030832 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:46.030805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:46.122807 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:46.122781    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"]
Jul 28 17:40:46.125189 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:40:46.125156    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0857cc69_854f_4f1c_a8de_33eb3b87062d.slice/crio-e6669bfdd91df40b10beba0842004f9d83af67f1d31db9ee29d380b913795e11 WatchSource:0}: Error finding container e6669bfdd91df40b10beba0842004f9d83af67f1d31db9ee29d380b913795e11: Status 404 returned error can't find the container with id e6669bfdd91df40b10beba0842004f9d83af67f1d31db9ee29d380b913795e11
Jul 28 17:40:46.608452 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:46.608395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" event={"ID":"0857cc69-854f-4f1c-a8de-33eb3b87062d","Type":"ContainerStarted","Data":"cfe494b64cbb533392c3386c802d3b3399758a9b6e0c1b3489290169870f1882"}
Jul 28 17:40:46.608648 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:46.608479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" event={"ID":"0857cc69-854f-4f1c-a8de-33eb3b87062d","Type":"ContainerStarted","Data":"e6669bfdd91df40b10beba0842004f9d83af67f1d31db9ee29d380b913795e11"}
Jul 28 17:40:48.259739 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:48.259699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:53.644528 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:53.644497    2579 generic.go:356] "Generic (PLEG): container finished" podID="0857cc69-854f-4f1c-a8de-33eb3b87062d" containerID="cfe494b64cbb533392c3386c802d3b3399758a9b6e0c1b3489290169870f1882" exitCode=0
Jul 28 17:40:53.644904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:53.644578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" event={"ID":"0857cc69-854f-4f1c-a8de-33eb3b87062d","Type":"ContainerDied","Data":"cfe494b64cbb533392c3386c802d3b3399758a9b6e0c1b3489290169870f1882"}
Jul 28 17:40:53.965902 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:53.965828    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:54.029799 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:54.029771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:54.650320 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:54.650285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" event={"ID":"0857cc69-854f-4f1c-a8de-33eb3b87062d","Type":"ContainerStarted","Data":"40e37924e4a9d14747ffc504427848ecf3fa327755701ece45caaef0a838bf29"}
Jul 28 17:40:54.673061 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:54.673013    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" podStartSLOduration=2.6236533829999997 podStartE2EDuration="9.672997316s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.049343933s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:54.670767508 +0000 UTC m=+1820.905732962" watchObservedRunningTime="2026-07-28 17:40:54.672997316 +0000 UTC m=+1820.907962767"
Jul 28 17:40:54.990619 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:54.990531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:55.990477 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:55.990440    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:56.002783 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:56.002759    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:56.670591 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:56.670558    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:40:58.669035 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:40:58.669001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:06.655178 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:06.655093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:06.671107 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:06.671065    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"]
Jul 28 17:42:06.681124 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:42:06.681092    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:42:06.681124 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:42:06.681124 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:42:06.681124 ip-10-0-137-238 kubenswrapper[2579]:  > containerID="40e37924e4a9d14747ffc504427848ecf3fa327755701ece45caaef0a838bf29" cmd=["/bin/sleep","15"]
Jul 28 17:42:06.681439 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:06.681146    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" message="" err=<
Jul 28 17:42:06.681439 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:42:06.681439 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:42:06.681439 ip-10-0-137-238 kubenswrapper[2579]:  >
Jul 28 17:42:06.681439 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:42:06.681165    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:42:06.681439 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:42:06.681439 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:42:06.681439 ip-10-0-137-238 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" podUID="0857cc69-854f-4f1c-a8de-33eb3b87062d" containerName="main" containerID="cri-o://40e37924e4a9d14747ffc504427848ecf3fa327755701ece45caaef0a838bf29"
Jul 28 17:42:06.681439 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:06.681215    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" podUID="0857cc69-854f-4f1c-a8de-33eb3b87062d" containerName="main" containerID="cri-o://40e37924e4a9d14747ffc504427848ecf3fa327755701ece45caaef0a838bf29" gracePeriod=60
Jul 28 17:42:06.986129 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:06.986095    2579 generic.go:356] "Generic (PLEG): container finished" podID="0857cc69-854f-4f1c-a8de-33eb3b87062d" containerID="40e37924e4a9d14747ffc504427848ecf3fa327755701ece45caaef0a838bf29" exitCode=0
Jul 28 17:42:06.986297 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:06.986163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" event={"ID":"0857cc69-854f-4f1c-a8de-33eb3b87062d","Type":"ContainerDied","Data":"40e37924e4a9d14747ffc504427848ecf3fa327755701ece45caaef0a838bf29"}
Jul 28 17:42:07.047936 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.047910    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:42:07.154352 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.154321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0857cc69-854f-4f1c-a8de-33eb3b87062d-tls-certs\") pod \"0857cc69-854f-4f1c-a8de-33eb3b87062d\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") "
Jul 28 17:42:07.154557 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.154368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfsjr\" (UniqueName: \"kubernetes.io/projected/0857cc69-854f-4f1c-a8de-33eb3b87062d-kube-api-access-pfsjr\") pod \"0857cc69-854f-4f1c-a8de-33eb3b87062d\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") "
Jul 28 17:42:07.154557 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.154454    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-dshm\") pod \"0857cc69-854f-4f1c-a8de-33eb3b87062d\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") "
Jul 28 17:42:07.154557 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.154477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-tmp-dir\") pod \"0857cc69-854f-4f1c-a8de-33eb3b87062d\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") "
Jul 28 17:42:07.154557 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.154514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-model-cache\") pod \"0857cc69-854f-4f1c-a8de-33eb3b87062d\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") "
Jul 28 17:42:07.154767 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.154623    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-home\") pod \"0857cc69-854f-4f1c-a8de-33eb3b87062d\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") "
Jul 28 17:42:07.154767 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.154653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-kserve-provision-location\") pod \"0857cc69-854f-4f1c-a8de-33eb3b87062d\" (UID: \"0857cc69-854f-4f1c-a8de-33eb3b87062d\") "
Jul 28 17:42:07.154767 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.154713    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0857cc69-854f-4f1c-a8de-33eb3b87062d" (UID: "0857cc69-854f-4f1c-a8de-33eb3b87062d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.154932 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.154804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0857cc69-854f-4f1c-a8de-33eb3b87062d" (UID: "0857cc69-854f-4f1c-a8de-33eb3b87062d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.156763 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.155524    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.156763 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.155560    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.156763 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.155580    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-home" (OuterVolumeSpecName: "home") pod "0857cc69-854f-4f1c-a8de-33eb3b87062d" (UID: "0857cc69-854f-4f1c-a8de-33eb3b87062d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.163596 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.163459    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-dshm" (OuterVolumeSpecName: "dshm") pod "0857cc69-854f-4f1c-a8de-33eb3b87062d" (UID: "0857cc69-854f-4f1c-a8de-33eb3b87062d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.163596 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.163462    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0857cc69-854f-4f1c-a8de-33eb3b87062d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0857cc69-854f-4f1c-a8de-33eb3b87062d" (UID: "0857cc69-854f-4f1c-a8de-33eb3b87062d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:07.164719 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.164693    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0857cc69-854f-4f1c-a8de-33eb3b87062d-kube-api-access-pfsjr" (OuterVolumeSpecName: "kube-api-access-pfsjr") pod "0857cc69-854f-4f1c-a8de-33eb3b87062d" (UID: "0857cc69-854f-4f1c-a8de-33eb3b87062d"). InnerVolumeSpecName "kube-api-access-pfsjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:07.216862 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.216780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0857cc69-854f-4f1c-a8de-33eb3b87062d" (UID: "0857cc69-854f-4f1c-a8de-33eb3b87062d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.256935 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.256902    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.256935 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.256933    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.257121 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.256943    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0857cc69-854f-4f1c-a8de-33eb3b87062d-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.257121 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.256953    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfsjr\" (UniqueName: \"kubernetes.io/projected/0857cc69-854f-4f1c-a8de-33eb3b87062d-kube-api-access-pfsjr\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.257121 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.256961    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0857cc69-854f-4f1c-a8de-33eb3b87062d-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.992543 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.992505    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn" event={"ID":"0857cc69-854f-4f1c-a8de-33eb3b87062d","Type":"ContainerDied","Data":"e6669bfdd91df40b10beba0842004f9d83af67f1d31db9ee29d380b913795e11"}
Jul 28 17:42:07.992543 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.992526    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"
Jul 28 17:42:07.993059 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:07.992567    2579 scope.go:117] "RemoveContainer" containerID="40e37924e4a9d14747ffc504427848ecf3fa327755701ece45caaef0a838bf29"
Jul 28 17:42:08.002738 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:08.002719    2579 scope.go:117] "RemoveContainer" containerID="cfe494b64cbb533392c3386c802d3b3399758a9b6e0c1b3489290169870f1882"
Jul 28 17:42:08.018985 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:08.018963    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"]
Jul 28 17:42:08.024509 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:08.024487    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-r47pn"]
Jul 28 17:42:08.099222 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:08.099192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:08.285049 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:08.284968    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0857cc69-854f-4f1c-a8de-33eb3b87062d" path="/var/lib/kubelet/pods/0857cc69-854f-4f1c-a8de-33eb3b87062d/volumes"
Jul 28 17:42:26.500361 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:26.500326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:26.596482 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:26.596446    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:32.712591 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:32.712553    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:36.752945 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.752906    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"]
Jul 28 17:42:36.753359 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.753317    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0857cc69-854f-4f1c-a8de-33eb3b87062d" containerName="main"
Jul 28 17:42:36.753359 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.753334    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0857cc69-854f-4f1c-a8de-33eb3b87062d" containerName="main"
Jul 28 17:42:36.753359 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.753350    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0857cc69-854f-4f1c-a8de-33eb3b87062d" containerName="storage-initializer"
Jul 28 17:42:36.753359 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.753360    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0857cc69-854f-4f1c-a8de-33eb3b87062d" containerName="storage-initializer"
Jul 28 17:42:36.753580 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.753457    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0857cc69-854f-4f1c-a8de-33eb3b87062d" containerName="main"
Jul 28 17:42:36.758012 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.757993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.760434 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.760395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 28 17:42:36.760549 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.760456    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 17:42:36.761313 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.761296    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 17:42:36.761483 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.761465    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 17:42:36.761540 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.761496    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-5rwls\""
Jul 28 17:42:36.763441 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.763419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:36.768898 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.768874    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"]
Jul 28 17:42:36.811593 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.811555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.811775 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.811640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.811775 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.811670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.811775 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.811714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.811775 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.811739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.811933 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.811804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9gv\" (UniqueName: \"kubernetes.io/projected/ce12b8ff-c162-4d04-84a5-2d916a547b09-kube-api-access-2h9gv\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.811933 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.811834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce12b8ff-c162-4d04-84a5-2d916a547b09-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913215 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913506 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913506 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9gv\" (UniqueName: \"kubernetes.io/projected/ce12b8ff-c162-4d04-84a5-2d916a547b09-kube-api-access-2h9gv\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce12b8ff-c162-4d04-84a5-2d916a547b09-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913628 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913783 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913783 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913711    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913783 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.913940 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.913922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.915815 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.915793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.916072 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.916054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce12b8ff-c162-4d04-84a5-2d916a547b09-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:36.921976 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:36.921951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9gv\" (UniqueName: \"kubernetes.io/projected/ce12b8ff-c162-4d04-84a5-2d916a547b09-kube-api-access-2h9gv\") pod \"tokenizer-migration-path-test-kserve-6666787596-8l75b\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:37.069844 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:37.069795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:37.137563 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:37.137525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:37.206867 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:37.206831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:37.217583 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:37.217560    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"]
Jul 28 17:42:37.219951 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:42:37.219930    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce12b8ff_c162_4d04_84a5_2d916a547b09.slice/crio-4e087c95755c0a4fc2e74a11e1a9cad0e9af2896c2e84fb25b3bc021e876463c WatchSource:0}: Error finding container 4e087c95755c0a4fc2e74a11e1a9cad0e9af2896c2e84fb25b3bc021e876463c: Status 404 returned error can't find the container with id 4e087c95755c0a4fc2e74a11e1a9cad0e9af2896c2e84fb25b3bc021e876463c
Jul 28 17:42:38.131248 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:38.131209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" event={"ID":"ce12b8ff-c162-4d04-84a5-2d916a547b09","Type":"ContainerStarted","Data":"767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4"}
Jul 28 17:42:38.131642 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:38.131274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" event={"ID":"ce12b8ff-c162-4d04-84a5-2d916a547b09","Type":"ContainerStarted","Data":"4e087c95755c0a4fc2e74a11e1a9cad0e9af2896c2e84fb25b3bc021e876463c"}
Jul 28 17:42:38.929837 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:38.929798    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:38.989170 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:38.989136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:46.908120 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:46.908088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:52.200802 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:52.200765    2579 generic.go:356] "Generic (PLEG): container finished" podID="ce12b8ff-c162-4d04-84a5-2d916a547b09" containerID="767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4" exitCode=0
Jul 28 17:42:52.201223 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:52.200813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" event={"ID":"ce12b8ff-c162-4d04-84a5-2d916a547b09","Type":"ContainerDied","Data":"767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4"}
Jul 28 17:42:53.207536 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:53.207499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" event={"ID":"ce12b8ff-c162-4d04-84a5-2d916a547b09","Type":"ContainerStarted","Data":"2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374"}
Jul 28 17:42:53.229296 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:53.229242    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" podStartSLOduration=2.854565557 podStartE2EDuration="17.229228449s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.374662892s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:53.227422353 +0000 UTC m=+1939.462387799" watchObservedRunningTime="2026-07-28 17:42:53.229228449 +0000 UTC m=+1939.464193901"
Jul 28 17:42:54.070670 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:54.070633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:57.070854 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:57.070810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:57.083655 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:57.083621    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:42:57.236512 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:42:57.236481    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:43:05.690612 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:05.690577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:05.749212 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:05.749177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:22.501035 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:22.501005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:22.507533 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:22.507505    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"]
Jul 28 17:43:22.516850 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:43:22.516801    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:43:22.516850 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:43:22.516850 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:43:22.516850 ip-10-0-137-238 kubenswrapper[2579]:  > containerID="2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374" cmd=["/bin/sleep","15"]
Jul 28 17:43:22.517087 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:22.516876    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" message="" err=<
Jul 28 17:43:22.517087 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:43:22.517087 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:43:22.517087 ip-10-0-137-238 kubenswrapper[2579]:  >
Jul 28 17:43:22.517087 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:43:22.516893    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:43:22.517087 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:43:22.517087 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:43:22.517087 ip-10-0-137-238 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" podUID="ce12b8ff-c162-4d04-84a5-2d916a547b09" containerName="main" containerID="cri-o://2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374"
Jul 28 17:43:22.517087 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:22.516937    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" podUID="ce12b8ff-c162-4d04-84a5-2d916a547b09" containerName="main" containerID="cri-o://2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374" gracePeriod=60
Jul 28 17:43:22.580032 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:22.580007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:22.873471 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:22.873446    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:43:23.021242 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021208    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-dshm\") pod \"ce12b8ff-c162-4d04-84a5-2d916a547b09\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") "
Jul 28 17:43:23.021423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-kserve-provision-location\") pod \"ce12b8ff-c162-4d04-84a5-2d916a547b09\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") "
Jul 28 17:43:23.021423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-tmp-dir\") pod \"ce12b8ff-c162-4d04-84a5-2d916a547b09\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") "
Jul 28 17:43:23.021423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-home\") pod \"ce12b8ff-c162-4d04-84a5-2d916a547b09\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") "
Jul 28 17:43:23.021423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h9gv\" (UniqueName: \"kubernetes.io/projected/ce12b8ff-c162-4d04-84a5-2d916a547b09-kube-api-access-2h9gv\") pod \"ce12b8ff-c162-4d04-84a5-2d916a547b09\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") "
Jul 28 17:43:23.021423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-model-cache\") pod \"ce12b8ff-c162-4d04-84a5-2d916a547b09\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") "
Jul 28 17:43:23.021423 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021364    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce12b8ff-c162-4d04-84a5-2d916a547b09-tls-certs\") pod \"ce12b8ff-c162-4d04-84a5-2d916a547b09\" (UID: \"ce12b8ff-c162-4d04-84a5-2d916a547b09\") "
Jul 28 17:43:23.021748 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021655    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce12b8ff-c162-4d04-84a5-2d916a547b09" (UID: "ce12b8ff-c162-4d04-84a5-2d916a547b09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.021748 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-home" (OuterVolumeSpecName: "home") pod "ce12b8ff-c162-4d04-84a5-2d916a547b09" (UID: "ce12b8ff-c162-4d04-84a5-2d916a547b09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.021748 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.021696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce12b8ff-c162-4d04-84a5-2d916a547b09" (UID: "ce12b8ff-c162-4d04-84a5-2d916a547b09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.023664 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.023636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce12b8ff-c162-4d04-84a5-2d916a547b09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce12b8ff-c162-4d04-84a5-2d916a547b09" (UID: "ce12b8ff-c162-4d04-84a5-2d916a547b09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:23.023765 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.023697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-dshm" (OuterVolumeSpecName: "dshm") pod "ce12b8ff-c162-4d04-84a5-2d916a547b09" (UID: "ce12b8ff-c162-4d04-84a5-2d916a547b09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.023912 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.023894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce12b8ff-c162-4d04-84a5-2d916a547b09-kube-api-access-2h9gv" (OuterVolumeSpecName: "kube-api-access-2h9gv") pod "ce12b8ff-c162-4d04-84a5-2d916a547b09" (UID: "ce12b8ff-c162-4d04-84a5-2d916a547b09"). InnerVolumeSpecName "kube-api-access-2h9gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:23.090932 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.090891    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce12b8ff-c162-4d04-84a5-2d916a547b09" (UID: "ce12b8ff-c162-4d04-84a5-2d916a547b09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.122275 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.122245    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.122275 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.122269    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.122478 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.122300    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.122478 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.122310    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.122478 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.122319    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9gv\" (UniqueName: \"kubernetes.io/projected/ce12b8ff-c162-4d04-84a5-2d916a547b09-kube-api-access-2h9gv\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.122478 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.122326    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce12b8ff-c162-4d04-84a5-2d916a547b09-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.122478 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.122334    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce12b8ff-c162-4d04-84a5-2d916a547b09-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.351044 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.350959    2579 generic.go:356] "Generic (PLEG): container finished" podID="ce12b8ff-c162-4d04-84a5-2d916a547b09" containerID="2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374" exitCode=0
Jul 28 17:43:23.351044 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.351023    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"
Jul 28 17:43:23.351225 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.351052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" event={"ID":"ce12b8ff-c162-4d04-84a5-2d916a547b09","Type":"ContainerDied","Data":"2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374"}
Jul 28 17:43:23.351225 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.351108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b" event={"ID":"ce12b8ff-c162-4d04-84a5-2d916a547b09","Type":"ContainerDied","Data":"4e087c95755c0a4fc2e74a11e1a9cad0e9af2896c2e84fb25b3bc021e876463c"}
Jul 28 17:43:23.351225 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.351131    2579 scope.go:117] "RemoveContainer" containerID="2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374"
Jul 28 17:43:23.360754 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.360739    2579 scope.go:117] "RemoveContainer" containerID="767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4"
Jul 28 17:43:23.368694 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.368676    2579 scope.go:117] "RemoveContainer" containerID="2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374"
Jul 28 17:43:23.368944 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:43:23.368915    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374\": container with ID starting with 2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374 not found: ID does not exist" containerID="2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374"
Jul 28 17:43:23.368996 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.368943    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374"} err="failed to get container status \"2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374\": rpc error: code = NotFound desc = could not find container \"2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374\": container with ID starting with 2e086c93532c3ba2670006da866d73005314ace661d88d0aa32fc189211f9374 not found: ID does not exist"
Jul 28 17:43:23.368996 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.368960    2579 scope.go:117] "RemoveContainer" containerID="767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4"
Jul 28 17:43:23.369205 ip-10-0-137-238 kubenswrapper[2579]: E0728 17:43:23.369187    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4\": container with ID starting with 767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4 not found: ID does not exist" containerID="767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4"
Jul 28 17:43:23.369244 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.369212    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4"} err="failed to get container status \"767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4\": rpc error: code = NotFound desc = could not find container \"767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4\": container with ID starting with 767c0bcd9e2c30eb5e8cdf9c0f680a54087d1e7b422c039b0c08dc465decc7e4 not found: ID does not exist"
Jul 28 17:43:23.373440 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.373387    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"]
Jul 28 17:43:23.377527 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.377507    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-8l75b"]
Jul 28 17:43:23.983621 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:23.983587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:24.286815 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:24.286735    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce12b8ff-c162-4d04-84a5-2d916a547b09" path="/var/lib/kubelet/pods/ce12b8ff-c162-4d04-84a5-2d916a547b09/volumes"
Jul 28 17:43:42.535459 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:42.535395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.630104 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:42.630064    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:49.265188 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:49.265149    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:54.779193 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:54.779158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:54.847759 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:43:54.847726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:04.137349 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:44:04.137312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:04.332106 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:44:04.332070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:08.048297 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:44:08.048256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:26.832354 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:44:26.832323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:37.358360 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:44:37.358324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:37.456921 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:44:37.456883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:57.087841 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:44:57.087731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:57.183233 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:44:57.183200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:58.546866 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:44:58.546833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:02.963883 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:02.963843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:08.750504 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:08.750470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:08.855430 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:08.855381    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:10.505780 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:10.505740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:10.777074 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:10.776987    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:15.491936 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:15.491899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:31.235572 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:31.235536    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6_auth-invalid-token-test-kserve-57cd4b8d86-7dpxq_d8797f78-2333-45d2-aabe-755fd05d755e/storage-initializer/0.log"
Jul 28 17:45:32.777018 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:32.776979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:32.877049 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:32.877004    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:39.060960 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.060919    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"]
Jul 28 17:45:39.061419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.061255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce12b8ff-c162-4d04-84a5-2d916a547b09" containerName="storage-initializer"
Jul 28 17:45:39.061419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.061267    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce12b8ff-c162-4d04-84a5-2d916a547b09" containerName="storage-initializer"
Jul 28 17:45:39.061419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.061276    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce12b8ff-c162-4d04-84a5-2d916a547b09" containerName="main"
Jul 28 17:45:39.061419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.061281    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce12b8ff-c162-4d04-84a5-2d916a547b09" containerName="main"
Jul 28 17:45:39.061419 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.061354    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce12b8ff-c162-4d04-84a5-2d916a547b09" containerName="main"
Jul 28 17:45:39.064456 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.064439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.066840 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.066819    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-8twpr\""
Jul 28 17:45:39.066954 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.066845    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Jul 28 17:45:39.066954 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.066886    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Jul 28 17:45:39.067085 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.067066    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Jul 28 17:45:39.067239 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.067223    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 28 17:45:39.077771 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.077749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"]
Jul 28 17:45:39.085526 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.085504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:39.191653 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.191622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-model-cache\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.191653 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.191654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931d65e6-a44b-4cea-84a8-fd8d268254b3-tls-certs\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.191861 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.191672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xh8k\" (UniqueName: \"kubernetes.io/projected/931d65e6-a44b-4cea-84a8-fd8d268254b3-kube-api-access-2xh8k\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.191861 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.191784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-kserve-provision-location\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.191861 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.191814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-dshm\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.191861 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.191834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-home\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.191992 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.191877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-tmp-dir\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.291073 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.291044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:39.293236 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-kserve-provision-location\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293298 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-dshm\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293298 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-home\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293298 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-tmp-dir\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-model-cache\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931d65e6-a44b-4cea-84a8-fd8d268254b3-tls-certs\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293444 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xh8k\" (UniqueName: \"kubernetes.io/projected/931d65e6-a44b-4cea-84a8-fd8d268254b3-kube-api-access-2xh8k\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293685 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-kserve-provision-location\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293786 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-tmp-dir\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293786 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-home\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.293874 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.293798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-model-cache\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.295559 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.295539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-dshm\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.295897 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.295880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931d65e6-a44b-4cea-84a8-fd8d268254b3-tls-certs\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.302168 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.302143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xh8k\" (UniqueName: \"kubernetes.io/projected/931d65e6-a44b-4cea-84a8-fd8d268254b3-kube-api-access-2xh8k\") pod \"auth-disabled-test-kserve-84794c675c-8w946\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.375247 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.375181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:39.508230 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.508205    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"]
Jul 28 17:45:39.510368 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:45:39.510317    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931d65e6_a44b_4cea_84a8_fd8d268254b3.slice/crio-f2ae284fd3168a2053dbadfc4b2eea5f6d32172a252ec837f0af6cf0075ab719 WatchSource:0}: Error finding container f2ae284fd3168a2053dbadfc4b2eea5f6d32172a252ec837f0af6cf0075ab719: Status 404 returned error can't find the container with id f2ae284fd3168a2053dbadfc4b2eea5f6d32172a252ec837f0af6cf0075ab719
Jul 28 17:45:39.975184 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.975146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" event={"ID":"931d65e6-a44b-4cea-84a8-fd8d268254b3","Type":"ContainerStarted","Data":"9aa5f9c078924a716c1256800e060ce4841088308777c524852b961104bb1608"}
Jul 28 17:45:39.975386 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:39.975205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" event={"ID":"931d65e6-a44b-4cea-84a8-fd8d268254b3","Type":"ContainerStarted","Data":"f2ae284fd3168a2053dbadfc4b2eea5f6d32172a252ec837f0af6cf0075ab719"}
Jul 28 17:45:45.287309 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:45.287276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:47.010351 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:47.010313    2579 generic.go:356] "Generic (PLEG): container finished" podID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerID="9aa5f9c078924a716c1256800e060ce4841088308777c524852b961104bb1608" exitCode=0
Jul 28 17:45:47.010923 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:47.010386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" event={"ID":"931d65e6-a44b-4cea-84a8-fd8d268254b3","Type":"ContainerDied","Data":"9aa5f9c078924a716c1256800e060ce4841088308777c524852b961104bb1608"}
Jul 28 17:45:48.016629 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:48.016597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" event={"ID":"931d65e6-a44b-4cea-84a8-fd8d268254b3","Type":"ContainerStarted","Data":"bce82430d9e29dacd1868b04829bb422471dff0e605844ce87ce13b09ebc8ccb"}
Jul 28 17:45:48.040443 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:48.040379    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podStartSLOduration=2.316371892 podStartE2EDuration="9.040363685s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.723991793s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:48.037585482 +0000 UTC m=+2114.272550946" watchObservedRunningTime="2026-07-28 17:45:48.040363685 +0000 UTC m=+2114.275329136"
Jul 28 17:45:48.375848 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:48.375806    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:49.375715 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:49.375676    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:45:49.377442 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:49.377390    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:45:59.376650 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:45:59.376594    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:46:00.539364 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:00.539322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:00.874870 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:00.874836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:05.333075 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:05.333030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:05.390071 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:05.390032    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:09.375773 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:09.375734    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:46:19.375525 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:19.375474    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:46:29.375943 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:29.375847    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:46:30.340270 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:30.340230    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:35.575872 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:35.575837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:35.658995 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:35.658953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:39.376077 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:39.376033    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:46:49.376345 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:49.376297    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:46:55.299552 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:55.299518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:55.371560 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:55.371524    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:59.376565 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:46:59.376514    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:47:09.375836 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:09.375782    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:47:15.310764 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:15.310711    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:16.824571 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:16.824535    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:19.376089 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:19.376034    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:47:19.988365 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:19.988328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:20.199728 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:20.199692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:23.242269 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:23.242229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:29.024216 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:29.024178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:29.127599 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:29.127569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:29.376674 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:29.376628    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:47:34.067351 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:34.067314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:39.386082 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:39.386051    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:47:39.393734 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:39.393707    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:47:43.970643 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:43.970604    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"]
Jul 28 17:47:43.979643 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:43.979615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:44.081936 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:44.081900    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:58.993650 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:58.993544    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" containerID="cri-o://bce82430d9e29dacd1868b04829bb422471dff0e605844ce87ce13b09ebc8ccb" gracePeriod=45
Jul 28 17:47:59.386932 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.386874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 28 17:47:59.673824 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.673783    2579 generic.go:356] "Generic (PLEG): container finished" podID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerID="bce82430d9e29dacd1868b04829bb422471dff0e605844ce87ce13b09ebc8ccb" exitCode=0
Jul 28 17:47:59.673975 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.673860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" event={"ID":"931d65e6-a44b-4cea-84a8-fd8d268254b3","Type":"ContainerDied","Data":"bce82430d9e29dacd1868b04829bb422471dff0e605844ce87ce13b09ebc8ccb"}
Jul 28 17:47:59.749353 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.749327    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:47:59.842348 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.842313    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-home\") pod \"931d65e6-a44b-4cea-84a8-fd8d268254b3\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") "
Jul 28 17:47:59.842348 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.842349    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-dshm\") pod \"931d65e6-a44b-4cea-84a8-fd8d268254b3\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") "
Jul 28 17:47:59.842606 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.842385    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-kserve-provision-location\") pod \"931d65e6-a44b-4cea-84a8-fd8d268254b3\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") "
Jul 28 17:47:59.842606 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.842440    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xh8k\" (UniqueName: \"kubernetes.io/projected/931d65e6-a44b-4cea-84a8-fd8d268254b3-kube-api-access-2xh8k\") pod \"931d65e6-a44b-4cea-84a8-fd8d268254b3\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") "
Jul 28 17:47:59.842606 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.842502    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931d65e6-a44b-4cea-84a8-fd8d268254b3-tls-certs\") pod \"931d65e6-a44b-4cea-84a8-fd8d268254b3\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") "
Jul 28 17:47:59.842606 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.842526    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-tmp-dir\") pod \"931d65e6-a44b-4cea-84a8-fd8d268254b3\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") "
Jul 28 17:47:59.842606 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.842589    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-model-cache\") pod \"931d65e6-a44b-4cea-84a8-fd8d268254b3\" (UID: \"931d65e6-a44b-4cea-84a8-fd8d268254b3\") "
Jul 28 17:47:59.842932 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.842904    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "931d65e6-a44b-4cea-84a8-fd8d268254b3" (UID: "931d65e6-a44b-4cea-84a8-fd8d268254b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:59.843154 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.843130    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-home" (OuterVolumeSpecName: "home") pod "931d65e6-a44b-4cea-84a8-fd8d268254b3" (UID: "931d65e6-a44b-4cea-84a8-fd8d268254b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:59.844925 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.844898    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931d65e6-a44b-4cea-84a8-fd8d268254b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "931d65e6-a44b-4cea-84a8-fd8d268254b3" (UID: "931d65e6-a44b-4cea-84a8-fd8d268254b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:59.845069 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.845046    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-dshm" (OuterVolumeSpecName: "dshm") pod "931d65e6-a44b-4cea-84a8-fd8d268254b3" (UID: "931d65e6-a44b-4cea-84a8-fd8d268254b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:59.845167 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.845146    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931d65e6-a44b-4cea-84a8-fd8d268254b3-kube-api-access-2xh8k" (OuterVolumeSpecName: "kube-api-access-2xh8k") pod "931d65e6-a44b-4cea-84a8-fd8d268254b3" (UID: "931d65e6-a44b-4cea-84a8-fd8d268254b3"). InnerVolumeSpecName "kube-api-access-2xh8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:59.860465 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.860392    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "931d65e6-a44b-4cea-84a8-fd8d268254b3" (UID: "931d65e6-a44b-4cea-84a8-fd8d268254b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:59.900484 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.900426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "931d65e6-a44b-4cea-84a8-fd8d268254b3" (UID: "931d65e6-a44b-4cea-84a8-fd8d268254b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:59.943904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.943832    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:47:59.943904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.943862    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:47:59.943904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.943871    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:47:59.943904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.943880    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:47:59.943904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.943890    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xh8k\" (UniqueName: \"kubernetes.io/projected/931d65e6-a44b-4cea-84a8-fd8d268254b3-kube-api-access-2xh8k\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:47:59.943904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.943898    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931d65e6-a44b-4cea-84a8-fd8d268254b3-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:47:59.943904 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:47:59.943907    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931d65e6-a44b-4cea-84a8-fd8d268254b3-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 17:48:00.680585 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:00.680542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946" event={"ID":"931d65e6-a44b-4cea-84a8-fd8d268254b3","Type":"ContainerDied","Data":"f2ae284fd3168a2053dbadfc4b2eea5f6d32172a252ec837f0af6cf0075ab719"}
Jul 28 17:48:00.680585 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:00.680566    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"
Jul 28 17:48:00.681121 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:00.680609    2579 scope.go:117] "RemoveContainer" containerID="bce82430d9e29dacd1868b04829bb422471dff0e605844ce87ce13b09ebc8ccb"
Jul 28 17:48:00.689256 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:00.689236    2579 scope.go:117] "RemoveContainer" containerID="9aa5f9c078924a716c1256800e060ce4841088308777c524852b961104bb1608"
Jul 28 17:48:00.699773 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:00.699745    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"]
Jul 28 17:48:00.705450 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:00.705424    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-84794c675c-8w946"]
Jul 28 17:48:02.284196 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:02.284161    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" path="/var/lib/kubelet/pods/931d65e6-a44b-4cea-84a8-fd8d268254b3/volumes"
Jul 28 17:48:03.991854 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:03.991821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:04.087784 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:04.087744    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:06.546860 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:06.546820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:06.817983 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:06.817893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:10.210869 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:10.210830    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:15.904842 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:15.904793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:16.005276 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:16.005241    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:17.048452 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.048418    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"]
Jul 28 17:48:17.048820 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.048810    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="storage-initializer"
Jul 28 17:48:17.048857 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.048822    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="storage-initializer"
Jul 28 17:48:17.048857 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.048843    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main"
Jul 28 17:48:17.048857 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.048850    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main"
Jul 28 17:48:17.048990 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.048932    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="931d65e6-a44b-4cea-84a8-fd8d268254b3" containerName="main"
Jul 28 17:48:17.051947 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.051930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.054526 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.054501    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-jt48n\""
Jul 28 17:48:17.055366 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.055347    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 28 17:48:17.055469 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.055433    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 28 17:48:17.055580 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.055559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 28 17:48:17.065744 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.065723    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"]
Jul 28 17:48:17.069626 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.069579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:17.191782 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.191742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krp6n\" (UniqueName: \"kubernetes.io/projected/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-kube-api-access-krp6n\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.192000 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.191850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.192000 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.191884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.192000 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.191913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.192000 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.191937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.192000 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.191985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.293546 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.293503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.293743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.293562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.293743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.293590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.293743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.293614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.293743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.293644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.293743 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.293718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krp6n\" (UniqueName: \"kubernetes.io/projected/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-kube-api-access-krp6n\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.293975 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.293856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.293975 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.293931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.294059 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.294028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.295860 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.295834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.296257 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.296238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.310373 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.310304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krp6n\" (UniqueName: \"kubernetes.io/projected/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-kube-api-access-krp6n\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.349375 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.349332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:17.362343 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.362317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:17.492230 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.492023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"]
Jul 28 17:48:17.495114 ip-10-0-137-238 kubenswrapper[2579]: W0728 17:48:17.495087    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb2a28b_3f58_47df_b39f_b6e5809d0e93.slice/crio-a95d8a6ed3f44d51e2647e4a01d221db3be52870907247793cb60a3431b0d1b1 WatchSource:0}: Error finding container a95d8a6ed3f44d51e2647e4a01d221db3be52870907247793cb60a3431b0d1b1: Status 404 returned error can't find the container with id a95d8a6ed3f44d51e2647e4a01d221db3be52870907247793cb60a3431b0d1b1
Jul 28 17:48:17.764969 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.764866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq" event={"ID":"3cb2a28b-3f58-47df-b39f-b6e5809d0e93","Type":"ContainerStarted","Data":"79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35"}
Jul 28 17:48:17.764969 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.764917    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq" event={"ID":"3cb2a28b-3f58-47df-b39f-b6e5809d0e93","Type":"ContainerStarted","Data":"a95d8a6ed3f44d51e2647e4a01d221db3be52870907247793cb60a3431b0d1b1"}
Jul 28 17:48:17.786386 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:17.786329    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq" podStartSLOduration=0.786311415 podStartE2EDuration="786.311415ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:17.784606193 +0000 UTC m=+2264.019571646" watchObservedRunningTime="2026-07-28 17:48:17.786311415 +0000 UTC m=+2264.021276866"
Jul 28 17:48:18.362683 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:18.362656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:27.363190 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:27.363148    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:27.375607 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:27.375580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:27.823803 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:27.823771    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 17:48:28.281352 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:28.281261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:44.559800 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:44.559768    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:48.918908 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:48.918866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:49.016966 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:48:49.016925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:08.422448 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:08.422379    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:08.519914 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:08.519877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:28.425122 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:28.425045    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:28.521789 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:28.521750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:33.178999 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:33.178965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:33.405949 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:33.405914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:35.523449 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:35.523390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:41.370189 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:41.370156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:41.441776 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:41.441723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:47.414882 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:47.414847    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:47.476266 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:47.476236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:49:51.859852 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:49:51.859816    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:02.744603 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:50:02.744570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:04.160761 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:04.160684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:04.261595 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:04.261565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:17.665108 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:17.665061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:17.755004 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:17.754972    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:37.627574 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:37.627539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:37.722992 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:37.722953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:44.373001 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:44.372952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:50.237913 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:50.237878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:50.319048 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:50.319017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:55.977473 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:55.977438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:56.355507 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:51:56.355472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:24.291612 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:52:24.291585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:17.893753 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:17.893713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:17.976910 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:17.976879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:37.416314 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:37.416279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:37.552632 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:37.552601    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:37.669315 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:37.669235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:37.765450 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:37.765421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:37.865543 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:37.865514    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:40.361410 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:40.361366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:40.430621 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:40.430589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:40.840296 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:40.840261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:43.949327 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:43.949295    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:49.545680 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:49.545639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:49.611168 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:53:49.611133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:04.728008 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:54:04.727904    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:10.379505 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:54:10.379469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:10.660003 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:54:10.659919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:17.555263 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:54:17.555229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:46.819569 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:55:46.819529    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:08.996706 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:08.996675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:09.067106 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:09.067071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:28.742828 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:28.742793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:28.832621 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:28.832588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:28.933809 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:28.933778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:29.035278 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:29.035199    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:29.105594 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:29.105566    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:30.164617 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:30.164527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:31.207532 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:31.207501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:31.308980 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:31.308942    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:32.060261 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:32.060228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:34.644465 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:34.644390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:34.709303 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:34.709276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:34.865705 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:34.865675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:34.967420 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:34.967322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:40.824594 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:40.824560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:40.924643 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:40.924612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:56.656129 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:56.656054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:56:56.878711 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:56:56.878644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:16.059290 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:57:16.059238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:47.923004 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:57:47.922971    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:00.500540 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:58:00.500499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:48.810203 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:59:48.810170    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:48.878495 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:59:48.878463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:56.014937 ip-10-0-137-238 kubenswrapper[2579]: I0728 17:59:56.014880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:08.756680 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:08.756646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:08.852350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:08.852314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:08.952173 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:08.952144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:09.052912 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:09.052885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:09.153689 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:09.153660    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:11.240887 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:11.240853    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:11.341597 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:11.341557    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:12.139312 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:12.139254    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:14.315525 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:14.315494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:14.416203 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:14.416174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:14.985988 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:14.985950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:17.520664 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:17.520627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:17.585310 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:17.585268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:20.765832 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:20.765794    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:20.833254 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:20.833225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:50.517433 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:50.517383    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:50.612983 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:50.612941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:59.631072 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:00:59.631036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:39.708787 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:02:39.708752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:44.566424 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:02:44.566375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:44.663529 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:02:44.663503    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:05.815543 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:05.815509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:06.285115 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.285037    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"]
Jul 28 18:03:06.293493 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:03:06.293458    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:03:06.293493 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:03:06.293493 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:03:06.293493 ip-10-0-137-238 kubenswrapper[2579]:  > containerID="79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35" cmd=["/bin/sleep","15"]
Jul 28 18:03:06.293797 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.293513    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq" message="" err=<
Jul 28 18:03:06.293797 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:03:06.293797 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:03:06.293797 ip-10-0-137-238 kubenswrapper[2579]:  >
Jul 28 18:03:06.293797 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:03:06.293529    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:03:06.293797 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:03:06.293797 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:03:06.293797 ip-10-0-137-238 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq" podUID="3cb2a28b-3f58-47df-b39f-b6e5809d0e93" containerName="main" containerID="cri-o://79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35"
Jul 28 18:03:06.293797 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.293577    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq" podUID="3cb2a28b-3f58-47df-b39f-b6e5809d0e93" containerName="main" containerID="cri-o://79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35" gracePeriod=60
Jul 28 18:03:06.628120 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.628096    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 18:03:06.763490 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.763462    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-dshm\") pod \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") "
Jul 28 18:03:06.763681 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.763499    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tls-certs\") pod \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") "
Jul 28 18:03:06.763681 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.763534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-home\") pod \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") "
Jul 28 18:03:06.763681 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.763651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krp6n\" (UniqueName: \"kubernetes.io/projected/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-kube-api-access-krp6n\") pod \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") "
Jul 28 18:03:06.763853 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.763732    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tmp-dir\") pod \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") "
Jul 28 18:03:06.763853 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.763784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-model-cache\") pod \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\" (UID: \"3cb2a28b-3f58-47df-b39f-b6e5809d0e93\") "
Jul 28 18:03:06.763853 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.763802    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-home" (OuterVolumeSpecName: "home") pod "3cb2a28b-3f58-47df-b39f-b6e5809d0e93" (UID: "3cb2a28b-3f58-47df-b39f-b6e5809d0e93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:06.764015 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.763989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3cb2a28b-3f58-47df-b39f-b6e5809d0e93" (UID: "3cb2a28b-3f58-47df-b39f-b6e5809d0e93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:06.764099 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.764076    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-model-cache" (OuterVolumeSpecName: "model-cache") pod "3cb2a28b-3f58-47df-b39f-b6e5809d0e93" (UID: "3cb2a28b-3f58-47df-b39f-b6e5809d0e93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:06.764159 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.764086    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:03:06.764159 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.764118    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:03:06.765898 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.765870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3cb2a28b-3f58-47df-b39f-b6e5809d0e93" (UID: "3cb2a28b-3f58-47df-b39f-b6e5809d0e93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:03:06.765898 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.765878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-kube-api-access-krp6n" (OuterVolumeSpecName: "kube-api-access-krp6n") pod "3cb2a28b-3f58-47df-b39f-b6e5809d0e93" (UID: "3cb2a28b-3f58-47df-b39f-b6e5809d0e93"). InnerVolumeSpecName "kube-api-access-krp6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:03:06.765898 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.765874    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-dshm" (OuterVolumeSpecName: "dshm") pod "3cb2a28b-3f58-47df-b39f-b6e5809d0e93" (UID: "3cb2a28b-3f58-47df-b39f-b6e5809d0e93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:06.864922 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.864866    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krp6n\" (UniqueName: \"kubernetes.io/projected/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-kube-api-access-krp6n\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:03:06.864922 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.864888    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:03:06.864922 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.864897    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:03:06.864922 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.864905    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cb2a28b-3f58-47df-b39f-b6e5809d0e93-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:03:06.911079 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.911052    2579 generic.go:356] "Generic (PLEG): container finished" podID="3cb2a28b-3f58-47df-b39f-b6e5809d0e93" containerID="79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35" exitCode=0
Jul 28 18:03:06.911193 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.911096    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"
Jul 28 18:03:06.911193 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.911139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq" event={"ID":"3cb2a28b-3f58-47df-b39f-b6e5809d0e93","Type":"ContainerDied","Data":"79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35"}
Jul 28 18:03:06.911279 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.911193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq" event={"ID":"3cb2a28b-3f58-47df-b39f-b6e5809d0e93","Type":"ContainerDied","Data":"a95d8a6ed3f44d51e2647e4a01d221db3be52870907247793cb60a3431b0d1b1"}
Jul 28 18:03:06.911279 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.911217    2579 scope.go:117] "RemoveContainer" containerID="79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35"
Jul 28 18:03:06.919672 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.919657    2579 scope.go:117] "RemoveContainer" containerID="79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35"
Jul 28 18:03:06.919901 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:03:06.919884    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35\": container with ID starting with 79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35 not found: ID does not exist" containerID="79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35"
Jul 28 18:03:06.919949 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.919910    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35"} err="failed to get container status \"79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35\": rpc error: code = NotFound desc = could not find container \"79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35\": container with ID starting with 79015f153dd009643c0368a6122434fc85d5b4ff42414ab6e1880af217f5ae35 not found: ID does not exist"
Jul 28 18:03:06.932143 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.932122    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"]
Jul 28 18:03:06.936095 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:06.936073    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-cbzdq"]
Jul 28 18:03:07.562856 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:07.562812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:08.192163 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:08.192118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:08.284298 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:08.284267    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb2a28b-3f58-47df-b39f-b6e5809d0e93" path="/var/lib/kubelet/pods/3cb2a28b-3f58-47df-b39f-b6e5809d0e93/volumes"
Jul 28 18:03:08.285812 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:08.285793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:11.913779 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:11.913714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:12.011768 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:12.011732    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:12.160667 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:12.160631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:12.259771 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:12.259666    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:12.361176 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:12.361139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:14.232888 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:14.232837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:14.301206 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:14.301172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:17.962619 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:17.962583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:18.061861 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:18.061836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:19.179744 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:19.179713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:19.257434 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:19.257385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:21.817450 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:21.817415    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:22.072799 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:22.072710    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:24.701278 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:24.701243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:24.771581 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:24.771519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:26.301435 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:26.301386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:26.404192 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:26.404164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:26.752786 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:26.752711    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:26.861620 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:26.861591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:26.965567 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:26.965538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:31.017045 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:31.017013    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:31.123265 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:31.123229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:31.218450 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:31.218395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:31.828415 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:31.828365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:34.776123 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:34.776030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:34.833561 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:34.833525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:35.871796 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:35.871740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:36.229865 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:36.229785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:36.320673 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:36.320642    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:38.230985 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:38.230953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:38.331000 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:38.330969    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:41.560411 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:41.560366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:41.662760 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:41.662721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:41.760569 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:41.760534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:42.468315 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:42.468278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:42.590054 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:42.590018    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:42.695225 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:42.695192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:43.025302 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:43.025274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:43.126388 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:43.126356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:43.228322 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:43.228291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:43.327943 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:43.327890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:44.547462 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:44.547428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:44.650042 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:44.650014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:44.751265 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:44.751235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:45.561773 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:45.561742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:45.664194 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:45.664155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:45.761942 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:45.761906    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:46.635567 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:46.635531    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:46.900565 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:46.900486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:48.469812 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:48.469775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:48.570328 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:48.570292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:49.411538 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:49.411504    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:49.511410 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:49.511376    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:49.753595 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:49.753521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:49.847289 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:49.847251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:50.951939 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:50.951910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:51.054541 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:51.054513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:52.237840 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:52.237811    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:52.340786 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:52.340757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:52.834449 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:52.834420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:52.934360 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:52.934332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:55.048668 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:03:55.048634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:02.363291 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.363259    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"]
Jul 28 18:04:02.363692 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.363636    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb2a28b-3f58-47df-b39f-b6e5809d0e93" containerName="main"
Jul 28 18:04:02.363692 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.363649    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb2a28b-3f58-47df-b39f-b6e5809d0e93" containerName="main"
Jul 28 18:04:02.363762 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.363708    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb2a28b-3f58-47df-b39f-b6e5809d0e93" containerName="main"
Jul 28 18:04:02.366764 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.366746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.369382 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.369360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-g9jzg\""
Jul 28 18:04:02.369863 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.369842    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 18:04:02.369940 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.369880    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 28 18:04:02.370803 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.370786    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 18:04:02.370880 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.370796    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 18:04:02.385993 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.385974    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"]
Jul 28 18:04:02.428352 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.428325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-home\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.428501 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.428357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-tmp-dir\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.428501 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.428393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdgkd\" (UniqueName: \"kubernetes.io/projected/db8e46f7-e3f7-4d04-866c-546009fd5967-kube-api-access-vdgkd\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.428501 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.428452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e46f7-e3f7-4d04-866c-546009fd5967-tls-certs\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.428501 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.428467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-dshm\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.428501 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.428498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-model-cache\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.428665 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.428558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-kserve-provision-location\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.472295 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.472243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:02.528877 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.528851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-kserve-provision-location\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.528990 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.528900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-home\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.529038 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.529017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-tmp-dir\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.529074 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.529059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgkd\" (UniqueName: \"kubernetes.io/projected/db8e46f7-e3f7-4d04-866c-546009fd5967-kube-api-access-vdgkd\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.529133 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.529116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e46f7-e3f7-4d04-866c-546009fd5967-tls-certs\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.529187 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.529149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-dshm\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.529187 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.529178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-model-cache\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.529297 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.529288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-home\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.529387 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.529362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-kserve-provision-location\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.529481 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.529439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-tmp-dir\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.529574 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.529559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-model-cache\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.531434 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.531387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-dshm\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.531684 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.531668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e46f7-e3f7-4d04-866c-546009fd5967-tls-certs\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.540066 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.540045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgkd\" (UniqueName: \"kubernetes.io/projected/db8e46f7-e3f7-4d04-866c-546009fd5967-kube-api-access-vdgkd\") pod \"stop-feature-test-kserve-65499d9cfd-586tw\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.677246 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.677173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:02.804490 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.804460    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:02.818122 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:02.818094    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"]
Jul 28 18:04:02.822368 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:04:02.822341    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8e46f7_e3f7_4d04_866c_546009fd5967.slice/crio-8be5a73f94b30bd54e40043bbc3b2669198851b7d58b3cae798ede1b6e4b8e62 WatchSource:0}: Error finding container 8be5a73f94b30bd54e40043bbc3b2669198851b7d58b3cae798ede1b6e4b8e62: Status 404 returned error can't find the container with id 8be5a73f94b30bd54e40043bbc3b2669198851b7d58b3cae798ede1b6e4b8e62
Jul 28 18:04:03.179491 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:03.179455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" event={"ID":"db8e46f7-e3f7-4d04-866c-546009fd5967","Type":"ContainerStarted","Data":"12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8"}
Jul 28 18:04:03.179688 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:03.179516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" event={"ID":"db8e46f7-e3f7-4d04-866c-546009fd5967","Type":"ContainerStarted","Data":"8be5a73f94b30bd54e40043bbc3b2669198851b7d58b3cae798ede1b6e4b8e62"}
Jul 28 18:04:07.643976 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:07.643938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:08.204980 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:08.204940    2579 generic.go:356] "Generic (PLEG): container finished" podID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerID="12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8" exitCode=0
Jul 28 18:04:08.205135 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:08.205014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" event={"ID":"db8e46f7-e3f7-4d04-866c-546009fd5967","Type":"ContainerDied","Data":"12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8"}
Jul 28 18:04:08.811010 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:08.810972    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:09.211887 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:09.211805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" event={"ID":"db8e46f7-e3f7-4d04-866c-546009fd5967","Type":"ContainerStarted","Data":"f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86"}
Jul 28 18:04:09.235332 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:09.235275    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podStartSLOduration=2.247136472 podStartE2EDuration="7.235257524s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.988121052s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:09.233746135 +0000 UTC m=+3215.468711598" watchObservedRunningTime="2026-07-28 18:04:09.235257524 +0000 UTC m=+3215.470222975"
Jul 28 18:04:09.677772 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:09.677733    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:12.678379 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:12.678335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:04:12.679873 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:12.679846    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:04:17.743080 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:17.743046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:22.678286 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:22.678245    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:04:32.678437 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:32.678332    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:04:42.678261 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:42.678209    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:04:51.332924 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:51.332888    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:51.433907 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:51.433873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:52.677735 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:04:52.677693    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:05:02.677590 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:05:02.677542    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:05:12.678470 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:05:12.678430    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:05:17.273158 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:05:17.273110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:22.678428 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:05:22.678379    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:05:32.677899 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:05:32.677850    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:05:42.677851 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:05:42.677807    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:05:52.678542 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:05:52.678497    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 28 18:06:02.687416 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:02.687370    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:06:02.695185 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:02.695159    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:06:04.817788 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:04.817754    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"]
Jul 28 18:06:06.308191 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:06.308155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:10.858863 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:10.858829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:11.038058 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:11.038024    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:18.108007 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:18.107966    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:18.395594 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:18.395502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:19.840084 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:19.840048    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main" containerID="cri-o://f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86" gracePeriod=45
Jul 28 18:06:20.594078 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.594054    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:06:20.756241 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.756154    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-model-cache\") pod \"db8e46f7-e3f7-4d04-866c-546009fd5967\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") "
Jul 28 18:06:20.756241 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.756210    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-dshm\") pod \"db8e46f7-e3f7-4d04-866c-546009fd5967\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") "
Jul 28 18:06:20.756514 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.756253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-tmp-dir\") pod \"db8e46f7-e3f7-4d04-866c-546009fd5967\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") "
Jul 28 18:06:20.756514 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.756279    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e46f7-e3f7-4d04-866c-546009fd5967-tls-certs\") pod \"db8e46f7-e3f7-4d04-866c-546009fd5967\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") "
Jul 28 18:06:20.756514 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.756320    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-home\") pod \"db8e46f7-e3f7-4d04-866c-546009fd5967\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") "
Jul 28 18:06:20.756514 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.756374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdgkd\" (UniqueName: \"kubernetes.io/projected/db8e46f7-e3f7-4d04-866c-546009fd5967-kube-api-access-vdgkd\") pod \"db8e46f7-e3f7-4d04-866c-546009fd5967\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") "
Jul 28 18:06:20.756514 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.756429    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-kserve-provision-location\") pod \"db8e46f7-e3f7-4d04-866c-546009fd5967\" (UID: \"db8e46f7-e3f7-4d04-866c-546009fd5967\") "
Jul 28 18:06:20.756514 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.756461    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-model-cache" (OuterVolumeSpecName: "model-cache") pod "db8e46f7-e3f7-4d04-866c-546009fd5967" (UID: "db8e46f7-e3f7-4d04-866c-546009fd5967"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:06:20.756801 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.756733    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:06:20.757516 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.757317    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-home" (OuterVolumeSpecName: "home") pod "db8e46f7-e3f7-4d04-866c-546009fd5967" (UID: "db8e46f7-e3f7-4d04-866c-546009fd5967"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:06:20.758868 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.758824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8e46f7-e3f7-4d04-866c-546009fd5967-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db8e46f7-e3f7-4d04-866c-546009fd5967" (UID: "db8e46f7-e3f7-4d04-866c-546009fd5967"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:06:20.758966 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.758862    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-dshm" (OuterVolumeSpecName: "dshm") pod "db8e46f7-e3f7-4d04-866c-546009fd5967" (UID: "db8e46f7-e3f7-4d04-866c-546009fd5967"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:06:20.759180 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.759149    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8e46f7-e3f7-4d04-866c-546009fd5967-kube-api-access-vdgkd" (OuterVolumeSpecName: "kube-api-access-vdgkd") pod "db8e46f7-e3f7-4d04-866c-546009fd5967" (UID: "db8e46f7-e3f7-4d04-866c-546009fd5967"). InnerVolumeSpecName "kube-api-access-vdgkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:06:20.774938 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.774910    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db8e46f7-e3f7-4d04-866c-546009fd5967" (UID: "db8e46f7-e3f7-4d04-866c-546009fd5967"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:06:20.821202 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.821174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db8e46f7-e3f7-4d04-866c-546009fd5967" (UID: "db8e46f7-e3f7-4d04-866c-546009fd5967"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:06:20.857120 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.857094    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdgkd\" (UniqueName: \"kubernetes.io/projected/db8e46f7-e3f7-4d04-866c-546009fd5967-kube-api-access-vdgkd\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:06:20.857120 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.857117    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:06:20.857601 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.857129    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:06:20.857601 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.857138    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:06:20.857601 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.857148    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8e46f7-e3f7-4d04-866c-546009fd5967-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:06:20.857601 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.857156    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8e46f7-e3f7-4d04-866c-546009fd5967-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:06:20.861376 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.861352    2579 generic.go:356] "Generic (PLEG): container finished" podID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerID="f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86" exitCode=0
Jul 28 18:06:20.861492 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.861423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"
Jul 28 18:06:20.861492 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.861440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" event={"ID":"db8e46f7-e3f7-4d04-866c-546009fd5967","Type":"ContainerDied","Data":"f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86"}
Jul 28 18:06:20.861581 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.861499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw" event={"ID":"db8e46f7-e3f7-4d04-866c-546009fd5967","Type":"ContainerDied","Data":"8be5a73f94b30bd54e40043bbc3b2669198851b7d58b3cae798ede1b6e4b8e62"}
Jul 28 18:06:20.861581 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.861527    2579 scope.go:117] "RemoveContainer" containerID="f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86"
Jul 28 18:06:20.870275 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.870257    2579 scope.go:117] "RemoveContainer" containerID="12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8"
Jul 28 18:06:20.878336 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.878321    2579 scope.go:117] "RemoveContainer" containerID="f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86"
Jul 28 18:06:20.878601 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:06:20.878582    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86\": container with ID starting with f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86 not found: ID does not exist" containerID="f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86"
Jul 28 18:06:20.878646 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.878608    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86"} err="failed to get container status \"f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86\": rpc error: code = NotFound desc = could not find container \"f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86\": container with ID starting with f2b20e2c30ee5451b2be70366886edc6351203ca8b30d85be93a789bdee70b86 not found: ID does not exist"
Jul 28 18:06:20.878646 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.878623    2579 scope.go:117] "RemoveContainer" containerID="12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8"
Jul 28 18:06:20.878822 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:06:20.878807    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8\": container with ID starting with 12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8 not found: ID does not exist" containerID="12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8"
Jul 28 18:06:20.878861 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.878825    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8"} err="failed to get container status \"12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8\": rpc error: code = NotFound desc = could not find container \"12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8\": container with ID starting with 12fe8ab445b5d4df0f12e56a63dfb369f9ebabea2744b6c97b1b0929cafb48d8 not found: ID does not exist"
Jul 28 18:06:20.884994 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.884963    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"]
Jul 28 18:06:20.889001 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:20.888980    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-586tw"]
Jul 28 18:06:22.285056 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:22.285015    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" path="/var/lib/kubelet/pods/db8e46f7-e3f7-4d04-866c-546009fd5967/volumes"
Jul 28 18:06:36.286171 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:36.286136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:44.274961 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:44.274929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:06:58.055813 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:06:58.055777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:29.816593 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:29.816553    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:30.822905 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:30.822866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:49.392917 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:49.392875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:49.491345 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:49.491300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:54.036230 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.036192    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"]
Jul 28 18:08:54.036641 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.036564    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="storage-initializer"
Jul 28 18:08:54.036641 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.036577    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="storage-initializer"
Jul 28 18:08:54.036641 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.036585    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main"
Jul 28 18:08:54.036641 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.036590    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main"
Jul 28 18:08:54.036641 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.036643    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="db8e46f7-e3f7-4d04-866c-546009fd5967" containerName="main"
Jul 28 18:08:54.039689 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.039670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.042269 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.042244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 28 18:08:54.042358 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.042302    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:08:54.042947 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.042928    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-2z74j\""
Jul 28 18:08:54.043037 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.042964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 28 18:08:54.043205 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.043186    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 28 18:08:54.043319 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.043301    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 28 18:08:54.054067 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.054045    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"]
Jul 28 18:08:54.064816 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.064791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:54.088617 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.088595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5prrl\" (UniqueName: \"kubernetes.io/projected/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kube-api-access-5prrl\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.088708 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.088630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-model-cache\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.088708 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.088683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.088797 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.088708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tls-certs\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.088797 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.088735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.088863 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.088801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tmp-dir\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.088863 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.088854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-home\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.088950 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.088870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-dshm\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190141 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-home\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190141 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-dshm\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190342 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5prrl\" (UniqueName: \"kubernetes.io/projected/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kube-api-access-5prrl\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190342 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-model-cache\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190554 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190554 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tls-certs\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190554 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190554 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tmp-dir\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190776 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-home\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190776 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-model-cache\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190776 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.190934 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.190832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tmp-dir\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.191239 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.191219    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.192498 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.192478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-dshm\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.192817 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.192798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tls-certs\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.198452 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.198429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5prrl\" (UniqueName: \"kubernetes.io/projected/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kube-api-access-5prrl\") pod \"lora-single-adapter-test-kserve-75845869bf-nvbjw\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.350294 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.350272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:08:54.489573 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.489532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"]
Jul 28 18:08:54.496880 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:08:54.496851    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0a9ea5_9712_4e48_a44c_e9e809d5fd19.slice/crio-5b18b22a276c2d11873b635419f2adde7de6a565f97ca70900bb6b8e83834e93 WatchSource:0}: Error finding container 5b18b22a276c2d11873b635419f2adde7de6a565f97ca70900bb6b8e83834e93: Status 404 returned error can't find the container with id 5b18b22a276c2d11873b635419f2adde7de6a565f97ca70900bb6b8e83834e93
Jul 28 18:08:54.553790 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:54.553762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" event={"ID":"be0a9ea5-9712-4e48-a44c-e9e809d5fd19","Type":"ContainerStarted","Data":"5b18b22a276c2d11873b635419f2adde7de6a565f97ca70900bb6b8e83834e93"}
Jul 28 18:08:55.559969 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:55.559929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" event={"ID":"be0a9ea5-9712-4e48-a44c-e9e809d5fd19","Type":"ContainerStarted","Data":"398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc"}
Jul 28 18:08:56.578089 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:08:56.578012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:09:02.078284 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:02.078245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:09:02.177187 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:02.177152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:09:04.602681 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:04.602647    2579 generic.go:356] "Generic (PLEG): container finished" podID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerID="398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc" exitCode=0
Jul 28 18:09:04.602681 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:04.602684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" event={"ID":"be0a9ea5-9712-4e48-a44c-e9e809d5fd19","Type":"ContainerDied","Data":"398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc"}
Jul 28 18:09:05.608885 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:05.608854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" event={"ID":"be0a9ea5-9712-4e48-a44c-e9e809d5fd19","Type":"ContainerStarted","Data":"6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e"}
Jul 28 18:09:05.633057 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:05.633007    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podStartSLOduration=2.3337803360000002 podStartE2EDuration="11.632991216s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.29921088s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:05.631464607 +0000 UTC m=+3511.866430060" watchObservedRunningTime="2026-07-28 18:09:05.632991216 +0000 UTC m=+3511.867956665"
Jul 28 18:09:06.351096 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:06.351063    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:09:14.350740 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:14.350703    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:09:14.351829 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:14.351803    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 28 18:09:16.123231 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:16.123198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:09:24.350849 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:24.350808    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 28 18:09:34.351422 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:34.351350    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 28 18:09:44.351059 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:44.351019    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 28 18:09:54.351249 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:09:54.351209    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 28 18:10:04.351472 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:10:04.351419    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 28 18:10:14.351221 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:10:14.351156    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 28 18:10:24.361251 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:10:24.361212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:10:24.369096 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:10:24.369072    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:10:59.078974 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:10:59.078939    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-75845869bf-nvbjw_be0a9ea5-9712-4e48-a44c-e9e809d5fd19/storage-initializer/0.log"
Jul 28 18:10:59.087291 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:10:59.087264    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-75845869bf-nvbjw_be0a9ea5-9712-4e48-a44c-e9e809d5fd19/main/0.log"
Jul 28 18:11:00.254710 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:00.254675    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"]
Jul 28 18:11:00.285952 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:00.285916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:15.276518 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.276458    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main" containerID="cri-o://6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e" gracePeriod=45
Jul 28 18:11:15.361978 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.361931    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 28 18:11:15.932946 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.932919    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:11:15.954991 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.954965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tls-certs\") pod \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") "
Jul 28 18:11:15.955156 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.955000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-cabundle-cert\") pod \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") "
Jul 28 18:11:15.955156 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.955061    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-dshm\") pod \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") "
Jul 28 18:11:15.955156 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.955116    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tmp-dir\") pod \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") "
Jul 28 18:11:15.955318 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.955299    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-home\") pod \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") "
Jul 28 18:11:15.955369 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.955332    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kserve-provision-location\") pod \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") "
Jul 28 18:11:15.955447 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.955366    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5prrl\" (UniqueName: \"kubernetes.io/projected/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kube-api-access-5prrl\") pod \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") "
Jul 28 18:11:15.955447 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.955391    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-model-cache\") pod \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\" (UID: \"be0a9ea5-9712-4e48-a44c-e9e809d5fd19\") "
Jul 28 18:11:15.955935 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.955846    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-home" (OuterVolumeSpecName: "home") pod "be0a9ea5-9712-4e48-a44c-e9e809d5fd19" (UID: "be0a9ea5-9712-4e48-a44c-e9e809d5fd19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:11:15.957040 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.957002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-model-cache" (OuterVolumeSpecName: "model-cache") pod "be0a9ea5-9712-4e48-a44c-e9e809d5fd19" (UID: "be0a9ea5-9712-4e48-a44c-e9e809d5fd19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:11:15.957426 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.957280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be0a9ea5-9712-4e48-a44c-e9e809d5fd19" (UID: "be0a9ea5-9712-4e48-a44c-e9e809d5fd19"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:11:15.958075 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.958042    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-dshm" (OuterVolumeSpecName: "dshm") pod "be0a9ea5-9712-4e48-a44c-e9e809d5fd19" (UID: "be0a9ea5-9712-4e48-a44c-e9e809d5fd19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:11:15.958765 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.958742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be0a9ea5-9712-4e48-a44c-e9e809d5fd19" (UID: "be0a9ea5-9712-4e48-a44c-e9e809d5fd19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:11:15.960300 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.960273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kube-api-access-5prrl" (OuterVolumeSpecName: "kube-api-access-5prrl") pod "be0a9ea5-9712-4e48-a44c-e9e809d5fd19" (UID: "be0a9ea5-9712-4e48-a44c-e9e809d5fd19"). InnerVolumeSpecName "kube-api-access-5prrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:11:15.974882 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:15.974822    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be0a9ea5-9712-4e48-a44c-e9e809d5fd19" (UID: "be0a9ea5-9712-4e48-a44c-e9e809d5fd19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:11:16.016799 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.016761    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be0a9ea5-9712-4e48-a44c-e9e809d5fd19" (UID: "be0a9ea5-9712-4e48-a44c-e9e809d5fd19"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:11:16.057127 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.057095    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:11:16.057127 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.057122    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:11:16.057127 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.057133    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:11:16.057341 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.057142    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:11:16.057341 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.057152    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5prrl\" (UniqueName: \"kubernetes.io/projected/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-kube-api-access-5prrl\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:11:16.057341 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.057163    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:11:16.057341 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.057172    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:11:16.057341 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.057179    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be0a9ea5-9712-4e48-a44c-e9e809d5fd19-cabundle-cert\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:11:16.276893 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.276802    2579 generic.go:356] "Generic (PLEG): container finished" podID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerID="6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e" exitCode=0
Jul 28 18:11:16.276893 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.276851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" event={"ID":"be0a9ea5-9712-4e48-a44c-e9e809d5fd19","Type":"ContainerDied","Data":"6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e"}
Jul 28 18:11:16.276893 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.276860    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"
Jul 28 18:11:16.277515 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.276927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw" event={"ID":"be0a9ea5-9712-4e48-a44c-e9e809d5fd19","Type":"ContainerDied","Data":"5b18b22a276c2d11873b635419f2adde7de6a565f97ca70900bb6b8e83834e93"}
Jul 28 18:11:16.277515 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.276951    2579 scope.go:117] "RemoveContainer" containerID="6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e"
Jul 28 18:11:16.285880 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.285852    2579 scope.go:117] "RemoveContainer" containerID="398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc"
Jul 28 18:11:16.294467 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.294444    2579 scope.go:117] "RemoveContainer" containerID="6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e"
Jul 28 18:11:16.294736 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:11:16.294717    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e\": container with ID starting with 6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e not found: ID does not exist" containerID="6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e"
Jul 28 18:11:16.294789 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.294746    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e"} err="failed to get container status \"6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e\": rpc error: code = NotFound desc = could not find container \"6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e\": container with ID starting with 6d3f79b9803a2ad384f7ed9941fc8a4a77ec09a41c9dd0777c092428df9f030e not found: ID does not exist"
Jul 28 18:11:16.294789 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.294765    2579 scope.go:117] "RemoveContainer" containerID="398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc"
Jul 28 18:11:16.295008 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:11:16.294989    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc\": container with ID starting with 398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc not found: ID does not exist" containerID="398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc"
Jul 28 18:11:16.295061 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.295014    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc"} err="failed to get container status \"398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc\": rpc error: code = NotFound desc = could not find container \"398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc\": container with ID starting with 398365f2e5f5646828af7c86515f119daa2322392caf5e42df47d9de81b784fc not found: ID does not exist"
Jul 28 18:11:16.301553 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.301531    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"]
Jul 28 18:11:16.304714 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:16.304693    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-nvbjw"]
Jul 28 18:11:18.285330 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:18.285292    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" path="/var/lib/kubelet/pods/be0a9ea5-9712-4e48-a44c-e9e809d5fd19/volumes"
Jul 28 18:11:20.375226 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:20.375195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:20.473344 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:20.473315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:20.572353 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:20.572322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:26.278157 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:26.278120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:26.339815 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:26.339781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:27.569073 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.569036    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"]
Jul 28 18:11:27.569456 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.569361    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="storage-initializer"
Jul 28 18:11:27.569456 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.569372    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="storage-initializer"
Jul 28 18:11:27.569456 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.569380    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main"
Jul 28 18:11:27.569456 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.569386    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main"
Jul 28 18:11:27.569602 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.569472    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0a9ea5-9712-4e48-a44c-e9e809d5fd19" containerName="main"
Jul 28 18:11:27.572532 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.572514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.575304 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.575275    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 28 18:11:27.575304 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.575283    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 28 18:11:27.575525 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.575316    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:11:27.575573 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.575541    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-25b9b\""
Jul 28 18:11:27.575627 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.575609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 28 18:11:27.576582 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.576560    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 28 18:11:27.585304 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.585282    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"]
Jul 28 18:11:27.601888 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.601864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:27.656931 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.656900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-home\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.657096 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.656937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.657096 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.657005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.657096 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.657049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-model-cache\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.657096 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.657068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.657096 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.657087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-dshm\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.657288 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.657112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.657288 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.657192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzxbl\" (UniqueName: \"kubernetes.io/projected/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kube-api-access-nzxbl\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758253 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758475 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-model-cache\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758475 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758475 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-dshm\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758631 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758631 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzxbl\" (UniqueName: \"kubernetes.io/projected/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kube-api-access-nzxbl\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758736 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-home\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758736 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758736 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-model-cache\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.758894 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.759002 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.758982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.759060 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.759000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-home\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.759108 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.759079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.760678 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.760660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-dshm\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.760986 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.760968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.766338 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.766316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzxbl\" (UniqueName: \"kubernetes.io/projected/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kube-api-access-nzxbl\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-zbc9m\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:27.883593 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:27.883501    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:28.022564 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:28.022345    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"]
Jul 28 18:11:28.024949 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:11:28.024917    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522e8b1d_6424_4cfd_bbd2_d53c9d1bb482.slice/crio-162d33bd94f1e5cc0a2eab5a1035ccc813709c81748437c2113671fa10cb195d WatchSource:0}: Error finding container 162d33bd94f1e5cc0a2eab5a1035ccc813709c81748437c2113671fa10cb195d: Status 404 returned error can't find the container with id 162d33bd94f1e5cc0a2eab5a1035ccc813709c81748437c2113671fa10cb195d
Jul 28 18:11:28.332906 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:28.332868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" event={"ID":"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482","Type":"ContainerStarted","Data":"0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c"}
Jul 28 18:11:28.333069 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:28.332937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" event={"ID":"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482","Type":"ContainerStarted","Data":"162d33bd94f1e5cc0a2eab5a1035ccc813709c81748437c2113671fa10cb195d"}
Jul 28 18:11:33.360561 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:33.360527    2579 generic.go:356] "Generic (PLEG): container finished" podID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerID="0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c" exitCode=0
Jul 28 18:11:33.360977 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:33.360580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" event={"ID":"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482","Type":"ContainerDied","Data":"0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c"}
Jul 28 18:11:34.367188 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:34.367146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" event={"ID":"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482","Type":"ContainerStarted","Data":"27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0"}
Jul 28 18:11:34.392854 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:34.392791    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" podStartSLOduration=2.265295408 podStartE2EDuration="7.392769812s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.127474404s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:11:34.390577738 +0000 UTC m=+3660.625543194" watchObservedRunningTime="2026-07-28 18:11:34.392769812 +0000 UTC m=+3660.627735264"
Jul 28 18:11:34.884640 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:34.884603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:37.884070 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:37.884018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:11:37.885709 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:37.885674    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 28 18:11:47.884150 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:47.884101    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 28 18:11:57.884286 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:11:57.884182    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 28 18:12:07.884575 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:12:07.884532    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 28 18:12:17.884002 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:12:17.883962    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 28 18:12:27.884452 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:12:27.884388    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 28 18:12:37.884517 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:12:37.884472    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 28 18:12:47.894831 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:12:47.894793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:12:47.903007 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:12:47.902982    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:13:31.748118 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:31.748042    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6d8b575478-zbc9m_522e8b1d-6424-4cfd-bbd2-d53c9d1bb482/storage-initializer/0.log"
Jul 28 18:13:31.760230 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:31.760205    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6d8b575478-zbc9m_522e8b1d-6424-4cfd-bbd2-d53c9d1bb482/main/0.log"
Jul 28 18:13:33.084900 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:33.084864    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"]
Jul 28 18:13:33.119866 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:33.119833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:48.106712 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.106654    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main" containerID="cri-o://27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0" gracePeriod=45
Jul 28 18:13:48.768766 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.768739    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:13:48.835280 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835248    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzxbl\" (UniqueName: \"kubernetes.io/projected/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kube-api-access-nzxbl\") pod \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") "
Jul 28 18:13:48.835481 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835306    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kserve-provision-location\") pod \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") "
Jul 28 18:13:48.835481 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835335    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-cabundle-cert\") pod \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") "
Jul 28 18:13:48.835481 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tls-certs\") pod \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") "
Jul 28 18:13:48.835481 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835424    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tmp-dir\") pod \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") "
Jul 28 18:13:48.835481 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835453    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-model-cache\") pod \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") "
Jul 28 18:13:48.835782 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835581    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-dshm\") pod \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") "
Jul 28 18:13:48.835782 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-home\") pod \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\" (UID: \"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482\") "
Jul 28 18:13:48.835782 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-model-cache" (OuterVolumeSpecName: "model-cache") pod "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" (UID: "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:13:48.835942 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.835782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" (UID: "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:13:48.836164 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.836106    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-cabundle-cert\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:13:48.836164 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.836131    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:13:48.836164 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.836153    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-home" (OuterVolumeSpecName: "home") pod "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" (UID: "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:13:48.837790 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.837758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" (UID: "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:13:48.837902 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.837797    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kube-api-access-nzxbl" (OuterVolumeSpecName: "kube-api-access-nzxbl") pod "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" (UID: "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482"). InnerVolumeSpecName "kube-api-access-nzxbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:13:48.837971 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.837953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-dshm" (OuterVolumeSpecName: "dshm") pod "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" (UID: "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:13:48.847874 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.847842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" (UID: "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:13:48.896682 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.896651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" (UID: "522e8b1d-6424-4cfd-bbd2-d53c9d1bb482"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:13:48.936570 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.936540    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:13:48.936570 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.936571    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:13:48.936570 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.936580    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:13:48.936713 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.936588    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:13:48.936713 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.936596    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzxbl\" (UniqueName: \"kubernetes.io/projected/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kube-api-access-nzxbl\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:13:48.936713 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:48.936605    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:13:49.065362 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.065327    2579 generic.go:356] "Generic (PLEG): container finished" podID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerID="27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0" exitCode=0
Jul 28 18:13:49.065532 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.065390    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"
Jul 28 18:13:49.065532 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.065429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" event={"ID":"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482","Type":"ContainerDied","Data":"27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0"}
Jul 28 18:13:49.065532 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.065491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m" event={"ID":"522e8b1d-6424-4cfd-bbd2-d53c9d1bb482","Type":"ContainerDied","Data":"162d33bd94f1e5cc0a2eab5a1035ccc813709c81748437c2113671fa10cb195d"}
Jul 28 18:13:49.065532 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.065521    2579 scope.go:117] "RemoveContainer" containerID="27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0"
Jul 28 18:13:49.074864 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.074847    2579 scope.go:117] "RemoveContainer" containerID="0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c"
Jul 28 18:13:49.083015 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.082998    2579 scope.go:117] "RemoveContainer" containerID="27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0"
Jul 28 18:13:49.083260 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:13:49.083240    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0\": container with ID starting with 27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0 not found: ID does not exist" containerID="27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0"
Jul 28 18:13:49.083345 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.083267    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0"} err="failed to get container status \"27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0\": rpc error: code = NotFound desc = could not find container \"27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0\": container with ID starting with 27e831863669b922dfad597734f791b9efde0cef38ca5295c364a8866fd089e0 not found: ID does not exist"
Jul 28 18:13:49.083345 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.083284    2579 scope.go:117] "RemoveContainer" containerID="0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c"
Jul 28 18:13:49.083553 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:13:49.083535    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c\": container with ID starting with 0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c not found: ID does not exist" containerID="0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c"
Jul 28 18:13:49.083603 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.083560    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c"} err="failed to get container status \"0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c\": rpc error: code = NotFound desc = could not find container \"0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c\": container with ID starting with 0f1f544ca95ea4dc9c7b77256f42e57b4f4f0e6dfc6b42325eedb66f3310f97c not found: ID does not exist"
Jul 28 18:13:49.099266 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.099218    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"]
Jul 28 18:13:49.104944 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:49.104915    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-zbc9m"]
Jul 28 18:13:50.285363 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:50.285323    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" path="/var/lib/kubelet/pods/522e8b1d-6424-4cfd-bbd2-d53c9d1bb482/volumes"
Jul 28 18:13:52.133658 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:52.133625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:53.296908 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:53.296873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:53.475763 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:53.475721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:58.668297 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.668258    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"]
Jul 28 18:13:58.668678 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.668606    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="storage-initializer"
Jul 28 18:13:58.668678 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.668618    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="storage-initializer"
Jul 28 18:13:58.668678 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.668628    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main"
Jul 28 18:13:58.668678 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.668634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main"
Jul 28 18:13:58.668825 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.668699    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="522e8b1d-6424-4cfd-bbd2-d53c9d1bb482" containerName="main"
Jul 28 18:13:58.671766 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.671750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.674592 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.674560    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 28 18:13:58.674932 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.674914    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-dzvkb\""
Jul 28 18:13:58.675128 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.675113    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 28 18:13:58.675419 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.675386    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 28 18:13:58.678993 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.678974    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 28 18:13:58.699065 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.699036    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"]
Jul 28 18:13:58.775935 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.775895    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:58.813064 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.813031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8866be8a-0140-4883-9910-27887e542c37-tls-certs\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.813216 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.813080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-home\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.813216 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.813141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-kserve-provision-location\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.813216 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.813193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-model-cache\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.813216 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.813214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-dshm\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.813448 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.813236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rjd\" (UniqueName: \"kubernetes.io/projected/8866be8a-0140-4883-9910-27887e542c37-kube-api-access-p7rjd\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.813448 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.813305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-tmp-dir\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914355 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8866be8a-0140-4883-9910-27887e542c37-tls-certs\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914579 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-home\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914579 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-kserve-provision-location\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914579 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-model-cache\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914579 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-dshm\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914579 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rjd\" (UniqueName: \"kubernetes.io/projected/8866be8a-0140-4883-9910-27887e542c37-kube-api-access-p7rjd\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914579 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-tmp-dir\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914900 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-home\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914900 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-model-cache\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914966 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-kserve-provision-location\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.914966 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.914934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-tmp-dir\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.916832 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.916798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-dshm\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.916954 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.916942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8866be8a-0140-4883-9910-27887e542c37-tls-certs\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.922318 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.922258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rjd\" (UniqueName: \"kubernetes.io/projected/8866be8a-0140-4883-9910-27887e542c37-kube-api-access-p7rjd\") pod \"tls-verification-test-kserve-55bf78c657-chq24\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:58.982141 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:58.982108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:13:59.055856 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:59.055824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:59.120313 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:59.120289    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"]
Jul 28 18:13:59.123003 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:13:59.122968    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8866be8a_0140_4883_9910_27887e542c37.slice/crio-eb841786e598dc8e02ba5ef9f751fa64fe058b2e207e7bc209bde0c58890340a WatchSource:0}: Error finding container eb841786e598dc8e02ba5ef9f751fa64fe058b2e207e7bc209bde0c58890340a: Status 404 returned error can't find the container with id eb841786e598dc8e02ba5ef9f751fa64fe058b2e207e7bc209bde0c58890340a
Jul 28 18:13:59.902320 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:13:59.902280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:14:00.001052 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:00.001016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:14:00.135087 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:00.135056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" event={"ID":"8866be8a-0140-4883-9910-27887e542c37","Type":"ContainerStarted","Data":"78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7"}
Jul 28 18:14:00.135243 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:00.135106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" event={"ID":"8866be8a-0140-4883-9910-27887e542c37","Type":"ContainerStarted","Data":"eb841786e598dc8e02ba5ef9f751fa64fe058b2e207e7bc209bde0c58890340a"}
Jul 28 18:14:09.180194 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:09.180157    2579 generic.go:356] "Generic (PLEG): container finished" podID="8866be8a-0140-4883-9910-27887e542c37" containerID="78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7" exitCode=0
Jul 28 18:14:09.180762 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:09.180233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" event={"ID":"8866be8a-0140-4883-9910-27887e542c37","Type":"ContainerDied","Data":"78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7"}
Jul 28 18:14:10.190110 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:10.190073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" event={"ID":"8866be8a-0140-4883-9910-27887e542c37","Type":"ContainerStarted","Data":"5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c"}
Jul 28 18:14:10.227122 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:10.227076    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podStartSLOduration=3.04431119 podStartE2EDuration="12.227060081s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.182748891s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:14:10.22515139 +0000 UTC m=+3816.460116841" watchObservedRunningTime="2026-07-28 18:14:10.227060081 +0000 UTC m=+3816.462025532"
Jul 28 18:14:10.983048 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:10.983014    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:14:12.893866 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:12.893827    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:14:18.982658 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:18.982619    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:14:18.984110 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:18.984083    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:14:28.983312 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:28.983261    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:14:38.983668 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:38.983620    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:14:48.982989 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:48.982945    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:14:58.983493 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:14:58.983450    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:15:08.982956 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:15:08.982910    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:15:18.983508 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:15:18.983458    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:15:28.982687 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:15:28.982636    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:15:38.670093 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:15:38.670053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:38.983615 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:15:38.983511    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:15:48.983135 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:15:48.983075    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:15:58.993153 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:15:58.993106    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:15:59.001555 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:15:59.001529    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:16:03.713870 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:03.713829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:03.714727 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:03.714703    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"]
Jul 28 18:16:03.780687 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:03.780654    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:18.737184 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:18.737120    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" containerID="cri-o://5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c" gracePeriod=45
Jul 28 18:16:18.994656 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:18.994560    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 28 18:16:19.403633 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.403606    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:16:19.421464 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.421439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7rjd\" (UniqueName: \"kubernetes.io/projected/8866be8a-0140-4883-9910-27887e542c37-kube-api-access-p7rjd\") pod \"8866be8a-0140-4883-9910-27887e542c37\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") "
Jul 28 18:16:19.421553 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.421487    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-dshm\") pod \"8866be8a-0140-4883-9910-27887e542c37\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") "
Jul 28 18:16:19.421553 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.421543    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8866be8a-0140-4883-9910-27887e542c37-tls-certs\") pod \"8866be8a-0140-4883-9910-27887e542c37\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") "
Jul 28 18:16:19.421622 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.421585    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-home\") pod \"8866be8a-0140-4883-9910-27887e542c37\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") "
Jul 28 18:16:19.421671 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.421624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-model-cache\") pod \"8866be8a-0140-4883-9910-27887e542c37\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") "
Jul 28 18:16:19.421671 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.421653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-tmp-dir\") pod \"8866be8a-0140-4883-9910-27887e542c37\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") "
Jul 28 18:16:19.421786 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.421713    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-kserve-provision-location\") pod \"8866be8a-0140-4883-9910-27887e542c37\" (UID: \"8866be8a-0140-4883-9910-27887e542c37\") "
Jul 28 18:16:19.421986 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.421924    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-model-cache" (OuterVolumeSpecName: "model-cache") pod "8866be8a-0140-4883-9910-27887e542c37" (UID: "8866be8a-0140-4883-9910-27887e542c37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:16:19.422415 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.422374    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-home" (OuterVolumeSpecName: "home") pod "8866be8a-0140-4883-9910-27887e542c37" (UID: "8866be8a-0140-4883-9910-27887e542c37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:16:19.423784 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.423764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-dshm" (OuterVolumeSpecName: "dshm") pod "8866be8a-0140-4883-9910-27887e542c37" (UID: "8866be8a-0140-4883-9910-27887e542c37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:16:19.423895 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.423878    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8866be8a-0140-4883-9910-27887e542c37-kube-api-access-p7rjd" (OuterVolumeSpecName: "kube-api-access-p7rjd") pod "8866be8a-0140-4883-9910-27887e542c37" (UID: "8866be8a-0140-4883-9910-27887e542c37"). InnerVolumeSpecName "kube-api-access-p7rjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:16:19.423961 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.423940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8866be8a-0140-4883-9910-27887e542c37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8866be8a-0140-4883-9910-27887e542c37" (UID: "8866be8a-0140-4883-9910-27887e542c37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:16:19.438199 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.438155    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8866be8a-0140-4883-9910-27887e542c37" (UID: "8866be8a-0140-4883-9910-27887e542c37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:16:19.487599 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.487542    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8866be8a-0140-4883-9910-27887e542c37" (UID: "8866be8a-0140-4883-9910-27887e542c37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:16:19.522950 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.522884    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:16:19.522950 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.522911    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7rjd\" (UniqueName: \"kubernetes.io/projected/8866be8a-0140-4883-9910-27887e542c37-kube-api-access-p7rjd\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:16:19.522950 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.522923    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:16:19.522950 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.522936    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8866be8a-0140-4883-9910-27887e542c37-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:16:19.522950 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.522944    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:16:19.522950 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.522952    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:16:19.523210 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.522960    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8866be8a-0140-4883-9910-27887e542c37-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:16:19.820635 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.820600    2579 generic.go:356] "Generic (PLEG): container finished" podID="8866be8a-0140-4883-9910-27887e542c37" containerID="5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c" exitCode=0
Jul 28 18:16:19.821057 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.820654    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"
Jul 28 18:16:19.821057 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.820679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" event={"ID":"8866be8a-0140-4883-9910-27887e542c37","Type":"ContainerDied","Data":"5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c"}
Jul 28 18:16:19.821057 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.820759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24" event={"ID":"8866be8a-0140-4883-9910-27887e542c37","Type":"ContainerDied","Data":"eb841786e598dc8e02ba5ef9f751fa64fe058b2e207e7bc209bde0c58890340a"}
Jul 28 18:16:19.821057 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.820783    2579 scope.go:117] "RemoveContainer" containerID="5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c"
Jul 28 18:16:19.830220 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.830200    2579 scope.go:117] "RemoveContainer" containerID="78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7"
Jul 28 18:16:19.840247 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.840225    2579 scope.go:117] "RemoveContainer" containerID="5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c"
Jul 28 18:16:19.840495 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:16:19.840475    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c\": container with ID starting with 5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c not found: ID does not exist" containerID="5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c"
Jul 28 18:16:19.840553 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.840503    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c"} err="failed to get container status \"5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c\": rpc error: code = NotFound desc = could not find container \"5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c\": container with ID starting with 5f4ae5031bf9c186ec1f8c35bb443ba1d27b88428f245d379a459cdd6c1fa30c not found: ID does not exist"
Jul 28 18:16:19.840553 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.840521    2579 scope.go:117] "RemoveContainer" containerID="78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7"
Jul 28 18:16:19.840786 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:16:19.840766    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7\": container with ID starting with 78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7 not found: ID does not exist" containerID="78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7"
Jul 28 18:16:19.840835 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.840793    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7"} err="failed to get container status \"78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7\": rpc error: code = NotFound desc = could not find container \"78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7\": container with ID starting with 78abf0eba6d0efa0d94ec108a2bcc180fd45883d0f916624c105d9258e2469f7 not found: ID does not exist"
Jul 28 18:16:19.844746 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.844722    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"]
Jul 28 18:16:19.851131 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:19.851109    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55bf78c657-chq24"]
Jul 28 18:16:20.285650 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:20.285573    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8866be8a-0140-4883-9910-27887e542c37" path="/var/lib/kubelet/pods/8866be8a-0140-4883-9910-27887e542c37/volumes"
Jul 28 18:16:23.614534 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:23.614491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:23.704997 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:23.704961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:29.541885 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:29.541792    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:35.168632 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:35.168595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:35.281280 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:35.281255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:36.466133 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.466096    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"]
Jul 28 18:16:36.466538 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.466434    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main"
Jul 28 18:16:36.466538 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.466444    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main"
Jul 28 18:16:36.466538 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.466467    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="storage-initializer"
Jul 28 18:16:36.466538 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.466472    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="storage-initializer"
Jul 28 18:16:36.466538 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.466530    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8866be8a-0140-4883-9910-27887e542c37" containerName="main"
Jul 28 18:16:36.470668 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.470646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.473287 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.473265    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 18:16:36.473453 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.473290    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 18:16:36.473453 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.473344    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 28 18:16:36.473642 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.473627    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 18:16:36.473693 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.473671    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-pqh2c\""
Jul 28 18:16:36.482968 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.482946    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"]
Jul 28 18:16:36.508674 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.508642    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:36.564386 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.564363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.564527 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.564461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.564527 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.564498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.564527 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.564521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.564667 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.564557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.564667 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.564650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.564733 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.564689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fdrl\" (UniqueName: \"kubernetes.io/projected/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kube-api-access-5fdrl\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.665471 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.665448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.665586 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.665478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fdrl\" (UniqueName: \"kubernetes.io/projected/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kube-api-access-5fdrl\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.665586 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.665503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.665717 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.665698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.665763 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.665730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.665763 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.665750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.665866 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.665799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.665974 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.665944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.666094 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.666042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.666094 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.666068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.666199 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.666123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.667681 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.667662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.668073 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.668058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.676614 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.676589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fdrl\" (UniqueName: \"kubernetes.io/projected/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kube-api-access-5fdrl\") pod \"prestop-hook-test-kserve-68f6bd7b79-ct989\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.728833 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.728752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:36.780883 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.780859    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:36.919160 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:36.919123    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"]
Jul 28 18:16:36.922385 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:16:36.922355    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8296db6_c1aa_449e_a6e6_fe7b7d5e4f86.slice/crio-6b5ea9402bd35b21d30d72b9a3674987871bd7515e578ed9b5e7d58f4e782b33 WatchSource:0}: Error finding container 6b5ea9402bd35b21d30d72b9a3674987871bd7515e578ed9b5e7d58f4e782b33: Status 404 returned error can't find the container with id 6b5ea9402bd35b21d30d72b9a3674987871bd7515e578ed9b5e7d58f4e782b33
Jul 28 18:16:37.906519 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:37.906486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" event={"ID":"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86","Type":"ContainerStarted","Data":"02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700"}
Jul 28 18:16:37.906974 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:37.906536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" event={"ID":"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86","Type":"ContainerStarted","Data":"6b5ea9402bd35b21d30d72b9a3674987871bd7515e578ed9b5e7d58f4e782b33"}
Jul 28 18:16:45.842420 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:45.842362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:46.954453 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:46.954417    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerID="02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700" exitCode=0
Jul 28 18:16:46.954453 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:46.954437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" event={"ID":"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86","Type":"ContainerDied","Data":"02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700"}
Jul 28 18:16:47.962369 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:47.962336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" event={"ID":"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86","Type":"ContainerStarted","Data":"98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2"}
Jul 28 18:16:47.987122 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:47.987069    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podStartSLOduration=2.492056625 podStartE2EDuration="11.98704946s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.494992835s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:16:47.985086841 +0000 UTC m=+3974.220052296" watchObservedRunningTime="2026-07-28 18:16:47.98704946 +0000 UTC m=+3974.222014913"
Jul 28 18:16:48.781741 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:48.781706    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:56.781160 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:56.781119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:16:56.782699 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:16:56.782670    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:17:06.782259 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:17:06.782215    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:17:16.781502 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:17:16.781460    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:17:26.781518 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:17:26.781467    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:17:36.782149 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:17:36.782102    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:17:46.782089 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:17:46.782035    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:17:56.781913 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:17:56.781816    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:18:06.782256 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:06.782207    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:18:08.160004 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:08.159965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:08.220204 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:08.220171    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:16.781522 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:16.781485    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:18:26.781491 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:26.781446    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:18:27.290885 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:27.290854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:27.651226 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:27.651189    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:36.791937 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:36.791900    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:18:36.799830 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:36.799804    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:18:39.393828 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:39.393791    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"]
Jul 28 18:18:47.349364 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:47.349326    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:49.478921 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:49.478886    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:54.415567 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:54.415527    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" containerID="cri-o://98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2" gracePeriod=45
Jul 28 18:18:54.793271 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:54.793175    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 28 18:18:55.087499 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.087476    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:18:55.179174 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179128    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-model-cache\") pod \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") "
Jul 28 18:18:55.179350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-home\") pod \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") "
Jul 28 18:18:55.179350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179220    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-dshm\") pod \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") "
Jul 28 18:18:55.179350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kserve-provision-location\") pod \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") "
Jul 28 18:18:55.179350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179287    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tls-certs\") pod \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") "
Jul 28 18:18:55.179350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tmp-dir\") pod \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") "
Jul 28 18:18:55.179639 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fdrl\" (UniqueName: \"kubernetes.io/projected/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kube-api-access-5fdrl\") pod \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\" (UID: \"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86\") "
Jul 28 18:18:55.179639 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179475    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" (UID: "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:55.179757 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179640    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:18:55.180005 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.179961    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-home" (OuterVolumeSpecName: "home") pod "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" (UID: "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:55.181508 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.181484    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-dshm" (OuterVolumeSpecName: "dshm") pod "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" (UID: "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:55.181673 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.181646    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" (UID: "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:18:55.181846 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.181823    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kube-api-access-5fdrl" (OuterVolumeSpecName: "kube-api-access-5fdrl") pod "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" (UID: "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86"). InnerVolumeSpecName "kube-api-access-5fdrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:18:55.191384 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.191352    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" (UID: "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:55.246461 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.246421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" (UID: "f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:55.280087 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.280056    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:18:55.280087 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.280084    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fdrl\" (UniqueName: \"kubernetes.io/projected/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kube-api-access-5fdrl\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:18:55.280087 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.280093    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:18:55.280283 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.280102    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:18:55.280283 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.280110    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:18:55.280283 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.280118    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:18:55.579717 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.579685    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerID="98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2" exitCode=0
Jul 28 18:18:55.580129 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.579735    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"
Jul 28 18:18:55.580129 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.579765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" event={"ID":"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86","Type":"ContainerDied","Data":"98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2"}
Jul 28 18:18:55.580129 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.579817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989" event={"ID":"f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86","Type":"ContainerDied","Data":"6b5ea9402bd35b21d30d72b9a3674987871bd7515e578ed9b5e7d58f4e782b33"}
Jul 28 18:18:55.580129 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.579839    2579 scope.go:117] "RemoveContainer" containerID="98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2"
Jul 28 18:18:55.589110 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.589095    2579 scope.go:117] "RemoveContainer" containerID="02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700"
Jul 28 18:18:55.597343 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.597324    2579 scope.go:117] "RemoveContainer" containerID="98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2"
Jul 28 18:18:55.597605 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:18:55.597586    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2\": container with ID starting with 98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2 not found: ID does not exist" containerID="98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2"
Jul 28 18:18:55.597660 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.597614    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2"} err="failed to get container status \"98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2\": rpc error: code = NotFound desc = could not find container \"98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2\": container with ID starting with 98cd5fcaffe76faf4bef3fc5fe669827a05ee958040c5b32d1725454780913d2 not found: ID does not exist"
Jul 28 18:18:55.597660 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.597630    2579 scope.go:117] "RemoveContainer" containerID="02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700"
Jul 28 18:18:55.597879 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:18:55.597858    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700\": container with ID starting with 02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700 not found: ID does not exist" containerID="02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700"
Jul 28 18:18:55.597971 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.597882    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700"} err="failed to get container status \"02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700\": rpc error: code = NotFound desc = could not find container \"02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700\": container with ID starting with 02e6c6e30c99e9d3f1742940cc0dd2a58d08e1ab7debc1aebc108fe81241a700 not found: ID does not exist"
Jul 28 18:18:55.602882 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.602861    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"]
Jul 28 18:18:55.606647 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:55.606627    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-ct989"]
Jul 28 18:18:56.284851 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:18:56.284821    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" path="/var/lib/kubelet/pods/f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86/volumes"
Jul 28 18:19:04.639298 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.639261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:04.739214 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.739177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:04.783744 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.783699    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:19:04.784054 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.784042    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="storage-initializer"
Jul 28 18:19:04.784107 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.784056    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="storage-initializer"
Jul 28 18:19:04.784107 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.784073    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main"
Jul 28 18:19:04.784107 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.784079    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main"
Jul 28 18:19:04.784213 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.784142    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8296db6-c1aa-449e-a6e6-fe7b7d5e4f86" containerName="main"
Jul 28 18:19:04.788588 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.788567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.792036 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.792007    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 28 18:19:04.792154 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.792067    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-rg2nd\""
Jul 28 18:19:04.792154 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.792083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 28 18:19:04.792154 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.792087    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 28 18:19:04.792154 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.792137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-nn5fq\""
Jul 28 18:19:04.798254 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.798233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:19:04.805485 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.805465    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:04.858676 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.858653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcr7z\" (UniqueName: \"kubernetes.io/projected/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-kube-api-access-rcr7z\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.858772 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.858680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.858772 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.858701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.858852 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.858793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.858852 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.858820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.858927 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.858867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.948713 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.948613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:04.959705 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.959683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.959785 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.959713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.959785 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.959742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.959785 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.959778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcr7z\" (UniqueName: \"kubernetes.io/projected/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-kube-api-access-rcr7z\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.959952 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.959796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.959952 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.959812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.960108 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.960088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.960188 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.960140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.960188 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.960177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.961893 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.961873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.961987 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.961973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:04.968559 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:04.968539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcr7z\" (UniqueName: \"kubernetes.io/projected/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-kube-api-access-rcr7z\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:05.099907 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:05.099871    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:05.237910 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:05.237888    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:19:05.239928 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:19:05.239901    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e07c83_9b40_4b2a_bdf6_b4ef6d526a04.slice/crio-97785ffbbf16489810bd8ce9889c64f7c7a6c040bee5dc6b0a3b6743b8acc458 WatchSource:0}: Error finding container 97785ffbbf16489810bd8ce9889c64f7c7a6c040bee5dc6b0a3b6743b8acc458: Status 404 returned error can't find the container with id 97785ffbbf16489810bd8ce9889c64f7c7a6c040bee5dc6b0a3b6743b8acc458
Jul 28 18:19:05.630960 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:05.630923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04","Type":"ContainerStarted","Data":"736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256"}
Jul 28 18:19:05.631133 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:05.630977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04","Type":"ContainerStarted","Data":"97785ffbbf16489810bd8ce9889c64f7c7a6c040bee5dc6b0a3b6743b8acc458"}
Jul 28 18:19:05.649515 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:05.649464    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podStartSLOduration=1.649449395 podStartE2EDuration="1.649449395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:19:05.647975934 +0000 UTC m=+4111.882941388" watchObservedRunningTime="2026-07-28 18:19:05.649449395 +0000 UTC m=+4111.884414847"
Jul 28 18:19:09.362320 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:09.362278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:09.428974 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:09.428932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:19.458126 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:19:19.458087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:13.037343 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:20:13.037303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:13.432900 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:20:13.432860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:17.721134 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:20:17.721095    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:17.941754 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:20:17.941725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:19.940520 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:20:19.940478    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:25.788919 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:20:25.788876    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:25.888685 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:20:25.888652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:20:37.541874 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:20:37.541829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:17.307721 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:21:17.307681    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:17.408523 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:21:17.408489    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:34.728522 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:21:34.728482    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:34.826790 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:21:34.826759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:34.930183 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:21:34.930148    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:41.057221 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:21:41.057184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:46.801919 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:21:46.801873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:46.898338 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:21:46.898297    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:59.560843 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:21:59.560804    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:00.493034 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:22:00.492995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:22.877327 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:22:22.877287    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:43.293690 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:22:43.293598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:43.748597 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:22:43.748556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:43.852225 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:22:43.852192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:43.955061 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:22:43.955030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:49.835785 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:22:49.835747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:49.908881 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:22:49.908850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:56.544851 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:22:56.544808    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:07.813439 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:23:07.813387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:30.637939 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:23:30.637902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:30.740354 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:23:30.740310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:30.838819 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:23:30.838790    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:34.995856 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:23:34.995822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:35.724623 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:23:35.724585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:44.542387 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:23:44.542357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:12.660860 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:24:12.660821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:13.848786 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:24:13.848749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:22.913994 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:24:22.913955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:27.610469 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:24:27.610434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:41.876617 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:24:41.876563    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:41.941752 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:24:41.941723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:46.782474 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:25:46.782376    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:46.897332 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:25:46.897300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:46.991396 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:25:46.991367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:51.838817 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:25:51.838783    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:25:55.041292 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:25:55.041237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:00.309863 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:00.309825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:00.410477 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:00.410444    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:06.689317 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:06.689282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:06.742463 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:06.742426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:28.581295 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:28.581257    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:28.636924 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:28.636890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:29.089299 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:29.089264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:32.689505 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:32.689469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:43.598137 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:43.598102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:43.665732 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:26:43.665702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:22.573535 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:27:22.573494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:22.673667 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:27:22.673625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:22.776530 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:27:22.776499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:27.090057 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:27:27.090025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:27.619921 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:27:27.619862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:39.345330 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:27:39.345286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:48.936115 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:27:48.936082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:49.037817 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:27:49.037784    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:52.665998 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:27:52.665963    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:04.048707 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:04.048670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:04.147999 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:04.147965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:04.760625 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:04.760585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:10.871191 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:10.871145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:10.941850 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:10.941822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:13.312704 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:13.312675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:29.286819 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:29.286738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:29.386318 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:29.386277    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:29.496536 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:29.496509    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:34.308096 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:34.308065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:37.943557 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:37.943522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:43.034992 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:28:43.034954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:10.732909 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:29:10.732871    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:11.540622 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:29:11.540586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:17.608138 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:29:17.608103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:23.226868 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:29:23.226831    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:23.327685 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:29:23.327655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:39.432127 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:29:39.432084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:05.863832 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:30:05.863798    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:05.968521 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:30:05.968486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:06.067803 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:30:06.067775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:10.933175 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:30:10.933134    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:12.136811 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:30:12.136777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:21.142241 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:30:21.142191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:47.781084 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:30:47.781048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:47.883415 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:30:47.883373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:48.296372 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:30:48.296341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:02.798081 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:02.798035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:02.911956 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:02.911930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:12.191323 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:12.191284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:34.910169 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:34.910083    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:35.005386 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:35.005353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:35.164033 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:35.163951    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:35.323062 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:35.323016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:39.206857 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:39.206823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:39.927369 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:39.927330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:47.603092 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:31:47.603059    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:32:16.150417 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:32:16.150363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:32:17.063173 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:32:17.063133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:32:31.583467 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:32:31.583432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:32:31.683252 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:32:31.683206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:01.381634 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:01.381540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:33.263719 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:33.263682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:38.267828 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:38.267791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:38.370926 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:38.370889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:43.309546 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:43.309511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:49.932245 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:49.932210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:50.597793 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:50.597760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:55.610662 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:55.610622    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:55.722750 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:55.722717    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:33:56.414029 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.413993    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:33:56.422812 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:33:56.422781    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:33:56.422812 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:33:56.422812 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:33:56.422812 ip-10-0-137-238 kubenswrapper[2579]:  > containerID="736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256" cmd=["/bin/sleep","15"]
Jul 28 18:33:56.423039 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.422829    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" message="" err=<
Jul 28 18:33:56.423039 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:33:56.423039 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:33:56.423039 ip-10-0-137-238 kubenswrapper[2579]:  >
Jul 28 18:33:56.423039 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:33:56.422841    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:33:56.423039 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:33:56.423039 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:33:56.423039 ip-10-0-137-238 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" containerName="main" containerID="cri-o://736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256"
Jul 28 18:33:56.423039 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.422882    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" containerName="main" containerID="cri-o://736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256" gracePeriod=60
Jul 28 18:33:56.666378 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.666319    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:33:56.746917 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.746886    2579 generic.go:356] "Generic (PLEG): container finished" podID="96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" containerID="736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256" exitCode=0
Jul 28 18:33:56.747038 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.746927    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:33:56.747038 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.746937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04","Type":"ContainerDied","Data":"736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256"}
Jul 28 18:33:56.747038 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.746977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04","Type":"ContainerDied","Data":"97785ffbbf16489810bd8ce9889c64f7c7a6c040bee5dc6b0a3b6743b8acc458"}
Jul 28 18:33:56.747038 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.747000    2579 scope.go:117] "RemoveContainer" containerID="736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256"
Jul 28 18:33:56.755508 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.755491    2579 scope.go:117] "RemoveContainer" containerID="736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256"
Jul 28 18:33:56.755740 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:33:56.755720    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256\": container with ID starting with 736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256 not found: ID does not exist" containerID="736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256"
Jul 28 18:33:56.755827 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.755745    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256"} err="failed to get container status \"736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256\": rpc error: code = NotFound desc = could not find container \"736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256\": container with ID starting with 736ac767958a0fd913db02a91b913523224140021dc43bbf395f8da815328256 not found: ID does not exist"
Jul 28 18:33:56.807633 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.807613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-model-cache\") pod \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") "
Jul 28 18:33:56.807732 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.807652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-home\") pod \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") "
Jul 28 18:33:56.807732 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.807669    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tls-certs\") pod \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") "
Jul 28 18:33:56.807732 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.807692    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-dshm\") pod \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") "
Jul 28 18:33:56.807876 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.807826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcr7z\" (UniqueName: \"kubernetes.io/projected/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-kube-api-access-rcr7z\") pod \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") "
Jul 28 18:33:56.807930 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.807884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tmp-dir\") pod \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\" (UID: \"96e07c83-9b40-4b2a-bdf6-b4ef6d526a04\") "
Jul 28 18:33:56.807930 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.807899    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-model-cache" (OuterVolumeSpecName: "model-cache") pod "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" (UID: "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:56.807930 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.807914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-home" (OuterVolumeSpecName: "home") pod "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" (UID: "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:56.808166 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.808140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" (UID: "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:56.808166 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.808146    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:33:56.808340 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.808182    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:33:56.809871 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.809843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-dshm" (OuterVolumeSpecName: "dshm") pod "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" (UID: "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:56.809970 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.809902    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" (UID: "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:33:56.809970 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.809945    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-kube-api-access-rcr7z" (OuterVolumeSpecName: "kube-api-access-rcr7z") pod "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" (UID: "96e07c83-9b40-4b2a-bdf6-b4ef6d526a04"). InnerVolumeSpecName "kube-api-access-rcr7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:33:56.909502 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.909473    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:33:56.909502 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.909497    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:33:56.909649 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.909512    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcr7z\" (UniqueName: \"kubernetes.io/projected/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-kube-api-access-rcr7z\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:33:56.909649 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:56.909551    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:33:57.068286 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:57.068253    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:33:57.071638 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:57.071615    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:33:58.284723 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:33:58.284688    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" path="/var/lib/kubelet/pods/96e07c83-9b40-4b2a-bdf6-b4ef6d526a04/volumes"
Jul 28 18:34:04.404194 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:04.404158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:05.266885 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:05.266787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:09.622382 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:09.622350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:11.093435 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:11.093377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:14.758012 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:14.757974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:14.858202 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:14.858169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:18.594477 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:18.594419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:24.370310 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.370272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:24.511026 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.510996    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 18:34:24.511429 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.511393    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" containerName="main"
Jul 28 18:34:24.511429 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.511429    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" containerName="main"
Jul 28 18:34:24.511561 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.511534    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="96e07c83-9b40-4b2a-bdf6-b4ef6d526a04" containerName="main"
Jul 28 18:34:24.514648 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.514628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.517249 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.517209    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 28 18:34:24.517388 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.517367    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 28 18:34:24.517463 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.517447    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 28 18:34:24.518389 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.518370    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-472qz\""
Jul 28 18:34:24.518480 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.518459    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-7gs5h\""
Jul 28 18:34:24.525839 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.525817    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 18:34:24.631306 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.631243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.631306 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.631292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.631474 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.631314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.631474 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.631380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.631474 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.631442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.631474 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.631465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdfgx\" (UniqueName: \"kubernetes.io/projected/dcdf6c46-402d-45a3-b43b-2533badb3f8d-kube-api-access-jdfgx\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.731920 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.731895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdfgx\" (UniqueName: \"kubernetes.io/projected/dcdf6c46-402d-45a3-b43b-2533badb3f8d-kube-api-access-jdfgx\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.732040 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.731942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.732040 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.731990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.732040 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.732014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.732162 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.732043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.732162 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.732066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.732439 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.732385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-home\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.732493 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.732445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.732493 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.732457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.734297 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.734270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-dshm\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.734429 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.734394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.740488 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.740468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdfgx\" (UniqueName: \"kubernetes.io/projected/dcdf6c46-402d-45a3-b43b-2533badb3f8d-kube-api-access-jdfgx\") pod \"autoscale-keda-lws-kserve-mn-0\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.826357 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.826332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:24.956600 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:24.956576    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 18:34:24.958223 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:34:24.958189    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdf6c46_402d_45a3_b43b_2533badb3f8d.slice/crio-63d4e04c9ce43fb6edaa8d0d5fa82261e02ac4af13d47d6b01cd893dcb5c2d07 WatchSource:0}: Error finding container 63d4e04c9ce43fb6edaa8d0d5fa82261e02ac4af13d47d6b01cd893dcb5c2d07: Status 404 returned error can't find the container with id 63d4e04c9ce43fb6edaa8d0d5fa82261e02ac4af13d47d6b01cd893dcb5c2d07
Jul 28 18:34:25.635075 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:25.635041    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:25.840229 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:25.840195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:25.889165 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:25.889068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"dcdf6c46-402d-45a3-b43b-2533badb3f8d","Type":"ContainerStarted","Data":"2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5"}
Jul 28 18:34:25.889165 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:25.889125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"dcdf6c46-402d-45a3-b43b-2533badb3f8d","Type":"ContainerStarted","Data":"63d4e04c9ce43fb6edaa8d0d5fa82261e02ac4af13d47d6b01cd893dcb5c2d07"}
Jul 28 18:34:25.910715 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:25.910665    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podStartSLOduration=1.910649007 podStartE2EDuration="1.910649007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:25.909486806 +0000 UTC m=+5032.144452260" watchObservedRunningTime="2026-07-28 18:34:25.910649007 +0000 UTC m=+5032.145614459"
Jul 28 18:34:26.826793 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:26.826759    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:30.105515 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:30.105470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:34.827393 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:34.827362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:34.839476 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:34.839453    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:34.946274 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:34.946246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:34:35.218958 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:35.218925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:35.319395 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:34:35.319358    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:55.314811 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:35:55.314771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:55.413509 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:35:55.413472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:14.443346 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:14.443311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:14.538480 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:14.538441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:17.106897 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:17.106851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:17.169935 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:17.169903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:20.524138 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:20.524103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:23.236954 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:23.236916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:23.506391 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:23.506301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:26.598492 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:26.598457    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:26.698456 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:26.698389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:35.587633 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:36:35.587597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:37:01.312276 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:37:01.312230    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:27.827236 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:38:27.827201    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:27.927492 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:38:27.927448    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:46.483645 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:38:46.483607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:47.816351 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:38:47.816318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:47.912933 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:38:47.912897    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:48.079684 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:38:48.079589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:48.183303 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:38:48.183271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:51.638718 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:38:51.638687    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:38:54.452184 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:38:54.452146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:00.122787 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:39:00.122702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:00.179849 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:39:00.179815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:12.862935 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:39:12.862894    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:13.416886 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:39:13.416852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:13.518623 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:39:13.518590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:40:26.783504 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:40:26.783419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:41:02.573843 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:41:02.573810    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:41:02.674082 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:41:02.674048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:43:33.677557 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:43:33.677521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:44:11.126141 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:44:11.126103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:31.840108 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:45:31.840069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:33.156477 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:45:33.156438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:51.826535 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:45:51.826498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:51.925044 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:45:51.925009    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:45:56.811417 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:45:56.811373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:08.820459 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:08.820420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:09.422090 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:09.422053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:18.222193 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:18.222159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:19.078786 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:19.078751    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:20.405215 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:20.405179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:24.639490 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:24.639453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:53.634258 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:53.634219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:55.393236 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:55.393205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:55.860180 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:55.860141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:46:55.959669 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:46:55.959625    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:16.036737 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.036698    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 18:49:16.045832 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:49:16.045796    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:49:16.045832 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:49:16.045832 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:49:16.045832 ip-10-0-137-238 kubenswrapper[2579]:  > containerID="2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5" cmd=["/bin/sleep","15"]
Jul 28 18:49:16.046086 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.045846    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" message="" err=<
Jul 28 18:49:16.046086 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:49:16.046086 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:49:16.046086 ip-10-0-137-238 kubenswrapper[2579]:  >
Jul 28 18:49:16.046086 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:49:16.045858    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:49:16.046086 ip-10-0-137-238 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:49:16.046086 ip-10-0-137-238 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:49:16.046086 ip-10-0-137-238 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="dcdf6c46-402d-45a3-b43b-2533badb3f8d" containerName="main" containerID="cri-o://2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5"
Jul 28 18:49:16.046086 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.045892    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" podUID="dcdf6c46-402d-45a3-b43b-2533badb3f8d" containerName="main" containerID="cri-o://2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5" gracePeriod=60
Jul 28 18:49:16.060551 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.060517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:16.118745 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.118715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:16.398199 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.398175    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:49:16.495667 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.495637    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-dshm\") pod \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") "
Jul 28 18:49:16.495842 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.495683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tls-certs\") pod \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") "
Jul 28 18:49:16.495842 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.495715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdfgx\" (UniqueName: \"kubernetes.io/projected/dcdf6c46-402d-45a3-b43b-2533badb3f8d-kube-api-access-jdfgx\") pod \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") "
Jul 28 18:49:16.495842 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.495747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tmp-dir\") pod \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") "
Jul 28 18:49:16.495974 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.495840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-model-cache\") pod \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") "
Jul 28 18:49:16.495974 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.495879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-home\") pod \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\" (UID: \"dcdf6c46-402d-45a3-b43b-2533badb3f8d\") "
Jul 28 18:49:16.496070 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.496004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dcdf6c46-402d-45a3-b43b-2533badb3f8d" (UID: "dcdf6c46-402d-45a3-b43b-2533badb3f8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:16.496118 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.496091    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "dcdf6c46-402d-45a3-b43b-2533badb3f8d" (UID: "dcdf6c46-402d-45a3-b43b-2533badb3f8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:16.496217 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.496199    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:49:16.496272 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.496222    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:49:16.496324 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.496294    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-home" (OuterVolumeSpecName: "home") pod "dcdf6c46-402d-45a3-b43b-2533badb3f8d" (UID: "dcdf6c46-402d-45a3-b43b-2533badb3f8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:16.498098 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.498075    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-dshm" (OuterVolumeSpecName: "dshm") pod "dcdf6c46-402d-45a3-b43b-2533badb3f8d" (UID: "dcdf6c46-402d-45a3-b43b-2533badb3f8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:16.498098 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.498085    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdf6c46-402d-45a3-b43b-2533badb3f8d-kube-api-access-jdfgx" (OuterVolumeSpecName: "kube-api-access-jdfgx") pod "dcdf6c46-402d-45a3-b43b-2533badb3f8d" (UID: "dcdf6c46-402d-45a3-b43b-2533badb3f8d"). InnerVolumeSpecName "kube-api-access-jdfgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:49:16.498253 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.498134    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcdf6c46-402d-45a3-b43b-2533badb3f8d" (UID: "dcdf6c46-402d-45a3-b43b-2533badb3f8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:49:16.597143 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.597050    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdfgx\" (UniqueName: \"kubernetes.io/projected/dcdf6c46-402d-45a3-b43b-2533badb3f8d-kube-api-access-jdfgx\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:49:16.597143 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.597081    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:49:16.597143 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.597094    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6c46-402d-45a3-b43b-2533badb3f8d-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:49:16.597143 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:16.597104    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdf6c46-402d-45a3-b43b-2533badb3f8d-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:49:17.015350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:17.015256    2579 generic.go:356] "Generic (PLEG): container finished" podID="dcdf6c46-402d-45a3-b43b-2533badb3f8d" containerID="2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5" exitCode=0
Jul 28 18:49:17.015350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:17.015294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"dcdf6c46-402d-45a3-b43b-2533badb3f8d","Type":"ContainerDied","Data":"2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5"}
Jul 28 18:49:17.015350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:17.015301    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"
Jul 28 18:49:17.015350 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:17.015338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0" event={"ID":"dcdf6c46-402d-45a3-b43b-2533badb3f8d","Type":"ContainerDied","Data":"63d4e04c9ce43fb6edaa8d0d5fa82261e02ac4af13d47d6b01cd893dcb5c2d07"}
Jul 28 18:49:17.015706 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:17.015360    2579 scope.go:117] "RemoveContainer" containerID="2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5"
Jul 28 18:49:17.025571 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:17.025550    2579 scope.go:117] "RemoveContainer" containerID="2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5"
Jul 28 18:49:17.025838 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:49:17.025820    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5\": container with ID starting with 2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5 not found: ID does not exist" containerID="2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5"
Jul 28 18:49:17.025888 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:17.025848    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5"} err="failed to get container status \"2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5\": rpc error: code = NotFound desc = could not find container \"2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5\": container with ID starting with 2d21b9258724cb84a46e583d00d2ec686b5e540c2e52fe4d90d0817981cdc1c5 not found: ID does not exist"
Jul 28 18:49:17.037726 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:17.037700    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 18:49:17.046812 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:17.046792    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0"]
Jul 28 18:49:18.285276 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:18.285243    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcdf6c46-402d-45a3-b43b-2533badb3f8d" path="/var/lib/kubelet/pods/dcdf6c46-402d-45a3-b43b-2533badb3f8d/volumes"
Jul 28 18:49:19.429748 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:19.429704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:19.501360 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:19.501329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:38.690131 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:38.690096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:38.790871 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:38.790838    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:44.651852 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:44.651814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:50.108569 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:50.108536    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:49:50.174500 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:49:50.174472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:50:04.660862 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.660824    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"]
Jul 28 18:50:04.661359 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.661284    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcdf6c46-402d-45a3-b43b-2533badb3f8d" containerName="main"
Jul 28 18:50:04.661359 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.661304    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcdf6c46-402d-45a3-b43b-2533badb3f8d" containerName="main"
Jul 28 18:50:04.661516 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.661465    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcdf6c46-402d-45a3-b43b-2533badb3f8d" containerName="main"
Jul 28 18:50:04.664667 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.664641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.672338 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.672315    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Jul 28 18:50:04.672472 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.672367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Jul 28 18:50:04.672472 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.672462    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zxjkm\""
Jul 28 18:50:04.672730 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.672713    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Jul 28 18:50:04.672779 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.672751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 28 18:50:04.703657 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.703635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"]
Jul 28 18:50:04.705878 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.705854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.705969 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.705953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28097526-e8f0-4932-83a9-6af24fd342de-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.706030 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.705986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.706030 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.706015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9zfj\" (UniqueName: \"kubernetes.io/projected/28097526-e8f0-4932-83a9-6af24fd342de-kube-api-access-n9zfj\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.706101 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.706046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.706101 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.706090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.706171 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.706126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.753095 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.753069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:50:04.807320 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807486 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807486 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807486 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28097526-e8f0-4932-83a9-6af24fd342de-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807486 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807723 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9zfj\" (UniqueName: \"kubernetes.io/projected/28097526-e8f0-4932-83a9-6af24fd342de-kube-api-access-n9zfj\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807723 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807832 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807891 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807891 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.807974 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.807953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.809810 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.809787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.810087 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.810068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28097526-e8f0-4932-83a9-6af24fd342de-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.819423 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.819385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9zfj\" (UniqueName: \"kubernetes.io/projected/28097526-e8f0-4932-83a9-6af24fd342de-kube-api-access-n9zfj\") pod \"custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:04.976159 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:04.976069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:05.005014 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:05.004982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:50:05.111887 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:05.111860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"]
Jul 28 18:50:05.113566 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:50:05.113536    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28097526_e8f0_4932_83a9_6af24fd342de.slice/crio-d42570e22314f5fb49e31f416cf8dabb3f3695437269198e1d1a700aacf20f96 WatchSource:0}: Error finding container d42570e22314f5fb49e31f416cf8dabb3f3695437269198e1d1a700aacf20f96: Status 404 returned error can't find the container with id d42570e22314f5fb49e31f416cf8dabb3f3695437269198e1d1a700aacf20f96
Jul 28 18:50:05.240701 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:05.240605    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" event={"ID":"28097526-e8f0-4932-83a9-6af24fd342de","Type":"ContainerStarted","Data":"e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21"}
Jul 28 18:50:05.240701 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:05.240669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" event={"ID":"28097526-e8f0-4932-83a9-6af24fd342de","Type":"ContainerStarted","Data":"d42570e22314f5fb49e31f416cf8dabb3f3695437269198e1d1a700aacf20f96"}
Jul 28 18:50:32.766148 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:32.766113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:50:34.374336 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:34.374305    2579 generic.go:356] "Generic (PLEG): container finished" podID="28097526-e8f0-4932-83a9-6af24fd342de" containerID="e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21" exitCode=0
Jul 28 18:50:34.374729 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:34.374379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" event={"ID":"28097526-e8f0-4932-83a9-6af24fd342de","Type":"ContainerDied","Data":"e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21"}
Jul 28 18:50:35.387873 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:35.387840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" event={"ID":"28097526-e8f0-4932-83a9-6af24fd342de","Type":"ContainerStarted","Data":"aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492"}
Jul 28 18:50:35.410316 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:35.410266    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podStartSLOduration=2.690145598 podStartE2EDuration="31.41024788s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.720102282s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:50:35.407942114 +0000 UTC m=+6001.642907565" watchObservedRunningTime="2026-07-28 18:50:35.41024788 +0000 UTC m=+6001.645213334"
Jul 28 18:50:44.976632 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:44.976582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:44.977191 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:44.976655    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:50:44.977970 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:44.977941    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:50:54.976980 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:50:54.976931    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:51:04.977109 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:51:04.977009    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:51:06.723615 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:51:06.723579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:51:14.976711 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:51:14.976660    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:51:24.976622 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:51:24.976577    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:51:34.976954 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:51:34.976905    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:51:44.977318 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:51:44.977276    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:51:54.976477 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:51:54.976430    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:52:04.976987 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:04.976932    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:52:14.976619 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:14.976572    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 28 18:52:24.986742 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:24.986713    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:52:24.994383 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:24.994357    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:52:43.322956 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:43.322873    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"]
Jul 28 18:52:43.369730 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:43.369701    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:43.455116 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:43.455084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:52:58.345929 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:58.345893    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main" containerID="cri-o://aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492" gracePeriod=45
Jul 28 18:52:59.003948 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.003924    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:52:59.069829 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.069793    2579 generic.go:356] "Generic (PLEG): container finished" podID="28097526-e8f0-4932-83a9-6af24fd342de" containerID="aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492" exitCode=0
Jul 28 18:52:59.069963 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.069847    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"
Jul 28 18:52:59.069963 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.069884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" event={"ID":"28097526-e8f0-4932-83a9-6af24fd342de","Type":"ContainerDied","Data":"aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492"}
Jul 28 18:52:59.069963 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.069940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx" event={"ID":"28097526-e8f0-4932-83a9-6af24fd342de","Type":"ContainerDied","Data":"d42570e22314f5fb49e31f416cf8dabb3f3695437269198e1d1a700aacf20f96"}
Jul 28 18:52:59.069963 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.069963    2579 scope.go:117] "RemoveContainer" containerID="aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492"
Jul 28 18:52:59.078185 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.078168    2579 scope.go:117] "RemoveContainer" containerID="e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21"
Jul 28 18:52:59.085828 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.085810    2579 scope.go:117] "RemoveContainer" containerID="aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492"
Jul 28 18:52:59.086062 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:52:59.086046    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492\": container with ID starting with aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492 not found: ID does not exist" containerID="aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492"
Jul 28 18:52:59.086111 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.086072    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492"} err="failed to get container status \"aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492\": rpc error: code = NotFound desc = could not find container \"aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492\": container with ID starting with aabd29806b388ff90503b667987944f9c017c842a530d536494e53f10ec4b492 not found: ID does not exist"
Jul 28 18:52:59.086111 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.086092    2579 scope.go:117] "RemoveContainer" containerID="e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21"
Jul 28 18:52:59.086288 ip-10-0-137-238 kubenswrapper[2579]: E0728 18:52:59.086272    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21\": container with ID starting with e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21 not found: ID does not exist" containerID="e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21"
Jul 28 18:52:59.086338 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.086293    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21"} err="failed to get container status \"e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21\": rpc error: code = NotFound desc = could not find container \"e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21\": container with ID starting with e59fe2360b6e547e95541fc204e72fc8b0879031ea330c07cc14600af1198a21 not found: ID does not exist"
Jul 28 18:52:59.122557 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.122536    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-home\") pod \"28097526-e8f0-4932-83a9-6af24fd342de\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") "
Jul 28 18:52:59.122656 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.122577    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9zfj\" (UniqueName: \"kubernetes.io/projected/28097526-e8f0-4932-83a9-6af24fd342de-kube-api-access-n9zfj\") pod \"28097526-e8f0-4932-83a9-6af24fd342de\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") "
Jul 28 18:52:59.122656 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.122594    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-dshm\") pod \"28097526-e8f0-4932-83a9-6af24fd342de\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") "
Jul 28 18:52:59.122656 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.122622    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28097526-e8f0-4932-83a9-6af24fd342de-tls-certs\") pod \"28097526-e8f0-4932-83a9-6af24fd342de\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") "
Jul 28 18:52:59.122656 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.122638    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-kserve-provision-location\") pod \"28097526-e8f0-4932-83a9-6af24fd342de\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") "
Jul 28 18:52:59.122874 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.122665    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-tmp-dir\") pod \"28097526-e8f0-4932-83a9-6af24fd342de\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") "
Jul 28 18:52:59.122874 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.122720    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-model-cache\") pod \"28097526-e8f0-4932-83a9-6af24fd342de\" (UID: \"28097526-e8f0-4932-83a9-6af24fd342de\") "
Jul 28 18:52:59.123196 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.123173    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-model-cache" (OuterVolumeSpecName: "model-cache") pod "28097526-e8f0-4932-83a9-6af24fd342de" (UID: "28097526-e8f0-4932-83a9-6af24fd342de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.123293 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.123265    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-home" (OuterVolumeSpecName: "home") pod "28097526-e8f0-4932-83a9-6af24fd342de" (UID: "28097526-e8f0-4932-83a9-6af24fd342de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.124941 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.124917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28097526-e8f0-4932-83a9-6af24fd342de-kube-api-access-n9zfj" (OuterVolumeSpecName: "kube-api-access-n9zfj") pod "28097526-e8f0-4932-83a9-6af24fd342de" (UID: "28097526-e8f0-4932-83a9-6af24fd342de"). InnerVolumeSpecName "kube-api-access-n9zfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:52:59.125055 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.124916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-dshm" (OuterVolumeSpecName: "dshm") pod "28097526-e8f0-4932-83a9-6af24fd342de" (UID: "28097526-e8f0-4932-83a9-6af24fd342de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.125055 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.124958    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28097526-e8f0-4932-83a9-6af24fd342de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28097526-e8f0-4932-83a9-6af24fd342de" (UID: "28097526-e8f0-4932-83a9-6af24fd342de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:52:59.134842 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.134816    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28097526-e8f0-4932-83a9-6af24fd342de" (UID: "28097526-e8f0-4932-83a9-6af24fd342de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.190360 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.190327    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28097526-e8f0-4932-83a9-6af24fd342de" (UID: "28097526-e8f0-4932-83a9-6af24fd342de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.223579 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.223552    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28097526-e8f0-4932-83a9-6af24fd342de-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.223579 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.223579    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.223725 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.223594    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.223725 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.223608    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.223725 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.223619    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.223725 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.223631    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9zfj\" (UniqueName: \"kubernetes.io/projected/28097526-e8f0-4932-83a9-6af24fd342de-kube-api-access-n9zfj\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.223725 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.223643    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28097526-e8f0-4932-83a9-6af24fd342de-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.392609 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.392577    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"]
Jul 28 18:52:59.397541 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:52:59.397517    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-58659d77fd-vdttx"]
Jul 28 18:53:00.285215 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:00.285178    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28097526-e8f0-4932-83a9-6af24fd342de" path="/var/lib/kubelet/pods/28097526-e8f0-4932-83a9-6af24fd342de/volumes"
Jul 28 18:53:03.466523 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:03.466490    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:03.567267 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:03.567236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:03.723472 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:03.723382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:03.823760 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:03.823731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:07.143933 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:07.143899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:09.990902 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:09.990869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:12.240582 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:12.240548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:15.677856 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:15.677817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:15.743260 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:15.743225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:26.091864 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.091816    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"]
Jul 28 18:53:26.092484 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.092364    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main"
Jul 28 18:53:26.092484 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.092383    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main"
Jul 28 18:53:26.092484 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.092396    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="storage-initializer"
Jul 28 18:53:26.092484 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.092416    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="storage-initializer"
Jul 28 18:53:26.092678 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.092500    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="28097526-e8f0-4932-83a9-6af24fd342de" containerName="main"
Jul 28 18:53:26.095957 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.095935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.098452 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.098427    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Jul 28 18:53:26.098589 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.098454    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Jul 28 18:53:26.098589 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.098468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 28 18:53:26.098589 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.098559    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Jul 28 18:53:26.098813 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.098796    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-rr7zn\""
Jul 28 18:53:26.109688 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.109663    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"]
Jul 28 18:53:26.124466 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.124442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:26.146791 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.146767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.146891 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.146813    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhmf\" (UniqueName: \"kubernetes.io/projected/d33c2b31-849b-4920-9249-8a793d222030-kube-api-access-7dhmf\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.146891 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.146866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-home\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.146891 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.146883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.147008 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.146898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.147008 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.146913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d33c2b31-849b-4920-9249-8a793d222030-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.147008 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.146929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.247441 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhmf\" (UniqueName: \"kubernetes.io/projected/d33c2b31-849b-4920-9249-8a793d222030-kube-api-access-7dhmf\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.247573 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-home\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.247573 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.247573 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.247573 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d33c2b31-849b-4920-9249-8a793d222030-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.247573 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.247811 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.247946 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.248092 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.248092 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.247982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.248092 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.248050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-home\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.249859 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.249839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.250094 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.250079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d33c2b31-849b-4920-9249-8a793d222030-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.254713 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.254687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhmf\" (UniqueName: \"kubernetes.io/projected/d33c2b31-849b-4920-9249-8a793d222030-kube-api-access-7dhmf\") pod \"router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.406639 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.406568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:26.544069 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.544039    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"]
Jul 28 18:53:26.545836 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:53:26.545810    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33c2b31_849b_4920_9249_8a793d222030.slice/crio-1a0cb648b384889048126a576c340a592f4ecdd51e64537086dd1544006d5e9b WatchSource:0}: Error finding container 1a0cb648b384889048126a576c340a592f4ecdd51e64537086dd1544006d5e9b: Status 404 returned error can't find the container with id 1a0cb648b384889048126a576c340a592f4ecdd51e64537086dd1544006d5e9b
Jul 28 18:53:26.710881 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.710799    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:26.770118 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:26.770089    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:53:27.206906 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:27.206866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" event={"ID":"d33c2b31-849b-4920-9249-8a793d222030","Type":"ContainerStarted","Data":"db6c68920ec92b5bec795ca26e213fd318275a7bf6aa28c860ae740b9cf86f54"}
Jul 28 18:53:27.207290 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:27.206927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" event={"ID":"d33c2b31-849b-4920-9249-8a793d222030","Type":"ContainerStarted","Data":"1a0cb648b384889048126a576c340a592f4ecdd51e64537086dd1544006d5e9b"}
Jul 28 18:53:34.243294 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:34.243265    2579 generic.go:356] "Generic (PLEG): container finished" podID="d33c2b31-849b-4920-9249-8a793d222030" containerID="db6c68920ec92b5bec795ca26e213fd318275a7bf6aa28c860ae740b9cf86f54" exitCode=0
Jul 28 18:53:34.243663 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:34.243332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" event={"ID":"d33c2b31-849b-4920-9249-8a793d222030","Type":"ContainerDied","Data":"db6c68920ec92b5bec795ca26e213fd318275a7bf6aa28c860ae740b9cf86f54"}
Jul 28 18:53:35.250258 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:35.250169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" event={"ID":"d33c2b31-849b-4920-9249-8a793d222030","Type":"ContainerStarted","Data":"1b2a14826688c9f7d490f0d252e7fc02997923eda72004579a047797413672ce"}
Jul 28 18:53:35.272916 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:35.272866    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podStartSLOduration=1.644175643 podStartE2EDuration="9.272851783s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.62867614s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:35.270941829 +0000 UTC m=+6181.505907280" watchObservedRunningTime="2026-07-28 18:53:35.272851783 +0000 UTC m=+6181.507817234"
Jul 28 18:53:36.407543 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:36.407503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:36.408033 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:36.407584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:53:36.408944 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:36.408916    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:53:46.407838 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:46.407792    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:53:56.407027 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:53:56.406939    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:54:06.406950 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:54:06.406912    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:54:16.407384 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:54:16.407340    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:54:23.910568 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:54:23.910528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:54:23.961660 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:54:23.961628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:54:26.407909 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:54:26.407859    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:54:36.407816 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:54:36.407766    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:54:46.407765 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:54:46.407722    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:54:56.407299 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:54:56.407248    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:55:06.407502 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:55:06.407459    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:55:16.407465 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:55:16.407392    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 28 18:55:26.416974 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:55:26.416895    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:55:26.424659 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:55:26.424632    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:55:45.342908 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:55:45.342873    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:55:45.353496 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:55:45.353462    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"]
Jul 28 18:55:45.437088 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:55:45.437054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:00.376743 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:00.376703    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main" containerID="cri-o://1b2a14826688c9f7d490f0d252e7fc02997923eda72004579a047797413672ce" gracePeriod=45
Jul 28 18:56:00.942598 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:00.942441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" event={"ID":"d33c2b31-849b-4920-9249-8a793d222030","Type":"ContainerDied","Data":"1b2a14826688c9f7d490f0d252e7fc02997923eda72004579a047797413672ce"}
Jul 28 18:56:00.942598 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:00.942471    2579 generic.go:356] "Generic (PLEG): container finished" podID="d33c2b31-849b-4920-9249-8a793d222030" containerID="1b2a14826688c9f7d490f0d252e7fc02997923eda72004579a047797413672ce" exitCode=0
Jul 28 18:56:01.019565 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.019537    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:56:01.111877 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.111809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-tmp-dir\") pod \"d33c2b31-849b-4920-9249-8a793d222030\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") "
Jul 28 18:56:01.111877 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.111846    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dhmf\" (UniqueName: \"kubernetes.io/projected/d33c2b31-849b-4920-9249-8a793d222030-kube-api-access-7dhmf\") pod \"d33c2b31-849b-4920-9249-8a793d222030\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") "
Jul 28 18:56:01.112086 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.111897    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-home\") pod \"d33c2b31-849b-4920-9249-8a793d222030\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") "
Jul 28 18:56:01.112086 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.111954    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-dshm\") pod \"d33c2b31-849b-4920-9249-8a793d222030\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") "
Jul 28 18:56:01.112086 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.111983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-model-cache\") pod \"d33c2b31-849b-4920-9249-8a793d222030\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") "
Jul 28 18:56:01.112086 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.112029    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-kserve-provision-location\") pod \"d33c2b31-849b-4920-9249-8a793d222030\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") "
Jul 28 18:56:01.112086 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.112052    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d33c2b31-849b-4920-9249-8a793d222030-tls-certs\") pod \"d33c2b31-849b-4920-9249-8a793d222030\" (UID: \"d33c2b31-849b-4920-9249-8a793d222030\") "
Jul 28 18:56:01.112351 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.112305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-model-cache" (OuterVolumeSpecName: "model-cache") pod "d33c2b31-849b-4920-9249-8a793d222030" (UID: "d33c2b31-849b-4920-9249-8a793d222030"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.112613 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.112591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-home" (OuterVolumeSpecName: "home") pod "d33c2b31-849b-4920-9249-8a793d222030" (UID: "d33c2b31-849b-4920-9249-8a793d222030"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.114284 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.114249    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33c2b31-849b-4920-9249-8a793d222030-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d33c2b31-849b-4920-9249-8a793d222030" (UID: "d33c2b31-849b-4920-9249-8a793d222030"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:56:01.114431 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.114412    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33c2b31-849b-4920-9249-8a793d222030-kube-api-access-7dhmf" (OuterVolumeSpecName: "kube-api-access-7dhmf") pod "d33c2b31-849b-4920-9249-8a793d222030" (UID: "d33c2b31-849b-4920-9249-8a793d222030"). InnerVolumeSpecName "kube-api-access-7dhmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:01.114506 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.114487    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-dshm" (OuterVolumeSpecName: "dshm") pod "d33c2b31-849b-4920-9249-8a793d222030" (UID: "d33c2b31-849b-4920-9249-8a793d222030"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.124869 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.124843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d33c2b31-849b-4920-9249-8a793d222030" (UID: "d33c2b31-849b-4920-9249-8a793d222030"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.176064 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.176027    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d33c2b31-849b-4920-9249-8a793d222030" (UID: "d33c2b31-849b-4920-9249-8a793d222030"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.212800 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.212778    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-kserve-provision-location\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.212800 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.212798    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d33c2b31-849b-4920-9249-8a793d222030-tls-certs\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.212921 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.212827    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-tmp-dir\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.212921 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.212836    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhmf\" (UniqueName: \"kubernetes.io/projected/d33c2b31-849b-4920-9249-8a793d222030-kube-api-access-7dhmf\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.212921 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.212844    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-home\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.212921 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.212853    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-dshm\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.212921 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.212861    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d33c2b31-849b-4920-9249-8a793d222030-model-cache\") on node \"ip-10-0-137-238.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.951810 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.951775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz" event={"ID":"d33c2b31-849b-4920-9249-8a793d222030","Type":"ContainerDied","Data":"1a0cb648b384889048126a576c340a592f4ecdd51e64537086dd1544006d5e9b"}
Jul 28 18:56:01.952232 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.951812    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"
Jul 28 18:56:01.952232 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.951842    2579 scope.go:117] "RemoveContainer" containerID="1b2a14826688c9f7d490f0d252e7fc02997923eda72004579a047797413672ce"
Jul 28 18:56:01.961419 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.961380    2579 scope.go:117] "RemoveContainer" containerID="db6c68920ec92b5bec795ca26e213fd318275a7bf6aa28c860ae740b9cf86f54"
Jul 28 18:56:01.974075 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.974052    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"]
Jul 28 18:56:01.977653 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:01.977631    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6996759666-5wdwz"]
Jul 28 18:56:02.285115 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:02.285039    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d33c2b31-849b-4920-9249-8a793d222030" path="/var/lib/kubelet/pods/d33c2b31-849b-4920-9249-8a793d222030/volumes"
Jul 28 18:56:05.389159 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:05.389127    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:08.124601 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:08.124569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:09.322365 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:09.322332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:09.716442 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:09.716391    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:10.438080 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:10.438048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:31.923631 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:31.923592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:32.987853 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:32.987822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:35.927332 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:35.927296    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:35.954247 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:35.954221    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:35.967000 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:35.966977    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:37.095507 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:37.095475    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:37.148484 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:37.148450    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:37.160990 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:37.160965    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:38.285269 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:38.285241    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:38.325542 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:38.325511    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:38.340716 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:38.340694    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:38.608461 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:38.608432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:39.492843 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:39.492807    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:39.531949 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:39.531920    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:39.544410 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:39.544381    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:40.643880 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:40.643847    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:40.684189 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:40.684163    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:40.697751 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:40.697727    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:41.803779 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:41.803742    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:41.841846 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:41.841819    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:41.854252 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:41.854229    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:42.967576 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:42.967545    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:43.008674 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:43.008644    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:43.021832 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:43.021809    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:44.131852 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:44.131812    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:44.171411 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:44.171373    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:44.183531 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:44.183506    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:45.281849 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:45.281816    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:45.349101 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:45.349066    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:45.361955 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:45.361928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:46.482706 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:46.482672    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:46.529922 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:46.529894    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:46.543080 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:46.543058    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:47.675249 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:47.675211    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:47.717170 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:47.717142    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:47.729438 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:47.729412    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:48.345999 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:48.345956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:48.861149 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:48.861114    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:48.902967 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:48.902941    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:48.915442 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:48.915419    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:50.041845 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:50.041810    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:56:50.086525 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:50.086495    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:56:50.107055 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:50.107028    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:56:52.372417 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:52.372380    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:52.519591 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:52.519556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:52.620903 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:52.620867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:52.903473 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:52.903438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:53.003723 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:53.003689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:55.128114 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:55.128078    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qf77t_b1684222-25b4-4a1c-b62f-df4ea459ffe9/global-pull-secret-syncer/0.log"
Jul 28 18:56:55.247593 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:55.247562    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log"
Jul 28 18:56:55.451605 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:55.451526    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-238.ec2.internal_6d68380c15b9e00b00440ef022ddd164/haproxy/0.log"
Jul 28 18:56:58.076432 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:58.076342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:56:58.143548 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:56:58.143515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bfd75_90388c56-3aae-4240-80a7-ed5560dcde24/konnectivity-agent/0.log" limit=1024
Jul 28 18:57:00.502104 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:00.502072    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-ft7f4_4358a29d-f0d8-48e1-ac74-a6e45464a8c0/authorino/0.log"
Jul 28 18:57:00.578028 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:00.577997    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-hsv2h_d736d71a-cd17-4573-8b79-365d3c6d9b85/manager/0.log"
Jul 28 18:57:00.602033 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:00.602008    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ffwbm_faf58333-3a4d-47ee-bd45-c944a60bc013/kuadrant-console-plugin/0.log"
Jul 28 18:57:02.551925 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:02.551892    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-klk6b_4b77928a-0f53-4b14-9cc0-02761f3299cb/kube-state-metrics/0.log"
Jul 28 18:57:02.572485 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:02.572457    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-klk6b_4b77928a-0f53-4b14-9cc0-02761f3299cb/kube-rbac-proxy-main/0.log"
Jul 28 18:57:02.593440 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:02.593395    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-klk6b_4b77928a-0f53-4b14-9cc0-02761f3299cb/kube-rbac-proxy-self/0.log"
Jul 28 18:57:02.634784 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:02.634764    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6788c89f75-58rcg_993984eb-49eb-4fc4-9c97-fdfcf3d07a7e/metrics-server/0.log"
Jul 28 18:57:02.713147 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:02.713126    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q5sz_f927c5f4-d7f2-47a3-8738-5520c084b7d7/node-exporter/0.log"
Jul 28 18:57:02.737391 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:02.737371    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q5sz_f927c5f4-d7f2-47a3-8738-5520c084b7d7/kube-rbac-proxy/0.log"
Jul 28 18:57:02.764845 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:02.764826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q5sz_f927c5f4-d7f2-47a3-8738-5520c084b7d7/init-textfile/0.log"
Jul 28 18:57:02.995978 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:02.995952    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mzwvz_9cc8b876-28fc-4f87-beb3-694d36ee6c14/kube-rbac-proxy-main/0.log"
Jul 28 18:57:03.021532 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:03.021505    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mzwvz_9cc8b876-28fc-4f87-beb3-694d36ee6c14/kube-rbac-proxy-self/0.log"
Jul 28 18:57:03.045177 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:03.045159    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mzwvz_9cc8b876-28fc-4f87-beb3-694d36ee6c14/openshift-state-metrics/0.log"
Jul 28 18:57:03.304687 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:03.304663    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-knbzg_6461a055-c2e9-4c93-a2b1-c2434ab41091/prometheus-operator-admission-webhook/0.log"
Jul 28 18:57:03.341660 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:03.341633    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f884997dd-7crk9_3af9ca0c-a879-42ff-aca1-e5c7fc8bac90/telemeter-client/0.log"
Jul 28 18:57:03.371227 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:03.371205    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f884997dd-7crk9_3af9ca0c-a879-42ff-aca1-e5c7fc8bac90/reload/0.log"
Jul 28 18:57:03.395412 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:03.395387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f884997dd-7crk9_3af9ca0c-a879-42ff-aca1-e5c7fc8bac90/kube-rbac-proxy/0.log"
Jul 28 18:57:04.708181 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:04.708147    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-fsxj8_628667c1-e4ab-4462-aceb-a83ef4a9c73b/networking-console-plugin/0.log"
Jul 28 18:57:05.672982 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.672953    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-657d64dcd5-52p67_d117fd2f-5e97-4b3c-ae7f-cd8f00b00638/console/0.log"
Jul 28 18:57:05.711701 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.711670    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-mr525_ed04b056-8c18-4a8e-8c9e-53bb72ed6c6a/download-server/0.log"
Jul 28 18:57:05.851883 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.851854    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"]
Jul 28 18:57:05.852250 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.852237    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main"
Jul 28 18:57:05.852298 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.852252    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main"
Jul 28 18:57:05.852298 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.852271    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="storage-initializer"
Jul 28 18:57:05.852298 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.852280    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="storage-initializer"
Jul 28 18:57:05.852395 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.852338    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d33c2b31-849b-4920-9249-8a793d222030" containerName="main"
Jul 28 18:57:05.855531 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.855498    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:05.857872 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.857850    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-67kr8\"/\"kube-root-ca.crt\""
Jul 28 18:57:05.858025 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.857938    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-67kr8\"/\"default-dockercfg-5smz9\""
Jul 28 18:57:05.858770 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.858744    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-67kr8\"/\"openshift-service-ca.crt\""
Jul 28 18:57:05.866506 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.866485    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"]
Jul 28 18:57:05.928076 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.928006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-885vj\" (UniqueName: \"kubernetes.io/projected/2852aec3-74f1-4e88-9129-272bc15daa20-kube-api-access-885vj\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:05.928076 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.928046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-sys\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:05.928247 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.928082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-proc\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:05.928247 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.928106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-lib-modules\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:05.928247 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:05.928136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-podres\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.028597 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.028566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-sys\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.028745 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.028625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-proc\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.028745 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.028651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-lib-modules\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.028745 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.028685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-sys\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.028745 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.028691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-podres\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.028924 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.028748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-proc\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.028924 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.028783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-885vj\" (UniqueName: \"kubernetes.io/projected/2852aec3-74f1-4e88-9129-272bc15daa20-kube-api-access-885vj\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.028924 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.028806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-podres\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.028924 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.028818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2852aec3-74f1-4e88-9129-272bc15daa20-lib-modules\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.036578 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.036554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-885vj\" (UniqueName: \"kubernetes.io/projected/2852aec3-74f1-4e88-9129-272bc15daa20-kube-api-access-885vj\") pod \"perf-node-gather-daemonset-r9kc5\" (UID: \"2852aec3-74f1-4e88-9129-272bc15daa20\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.165726 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.165699    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:06.296312 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:06.296283    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"]
Jul 28 18:57:06.298740 ip-10-0-137-238 kubenswrapper[2579]: W0728 18:57:06.298710    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2852aec3_74f1_4e88_9129_272bc15daa20.slice/crio-8f8e16de9419c43f8e6df30b8624aa7c61e8d21c00e78952dfe817c6d8aa2be5 WatchSource:0}: Error finding container 8f8e16de9419c43f8e6df30b8624aa7c61e8d21c00e78952dfe817c6d8aa2be5: Status 404 returned error can't find the container with id 8f8e16de9419c43f8e6df30b8624aa7c61e8d21c00e78952dfe817c6d8aa2be5
Jul 28 18:57:07.263185 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:07.263151    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5" event={"ID":"2852aec3-74f1-4e88-9129-272bc15daa20","Type":"ContainerStarted","Data":"378d166e8a48f66224a470e9714dcbd523a88ad5754cdf5e9b69086bc5db7d21"}
Jul 28 18:57:07.263617 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:07.263203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5" event={"ID":"2852aec3-74f1-4e88-9129-272bc15daa20","Type":"ContainerStarted","Data":"8f8e16de9419c43f8e6df30b8624aa7c61e8d21c00e78952dfe817c6d8aa2be5"}
Jul 28 18:57:07.263617 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:07.263293    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:07.280646 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:07.280592    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5" podStartSLOduration=2.280578283 podStartE2EDuration="2.280578283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:57:07.279203137 +0000 UTC m=+6393.514168589" watchObservedRunningTime="2026-07-28 18:57:07.280578283 +0000 UTC m=+6393.515543736"
Jul 28 18:57:07.328387 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:07.328362    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5g4wm_f1dd19dd-f660-4f6e-b57e-3de8377107b8/dns/0.log"
Jul 28 18:57:07.350209 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:07.350183    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5g4wm_f1dd19dd-f660-4f6e-b57e-3de8377107b8/kube-rbac-proxy/0.log"
Jul 28 18:57:07.465055 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:07.465020    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7tk7q_b1da3678-dfa2-4009-8d40-f74dc4c40c68/dns-node-resolver/0.log"
Jul 28 18:57:08.028602 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:08.028554    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-776cdb854c-qdpln_52f18dc8-7cba-445b-b8d8-a1b68bfe7d37/registry/0.log"
Jul 28 18:57:08.108206 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:08.108167    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qkksk_8dd162ac-a8b4-4ed7-9537-e37504a11570/node-ca/0.log"
Jul 28 18:57:09.988878 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:09.988848    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dxmmk_0f06086f-a8d1-4386-9075-165e5a7ee8b5/serve-healthcheck-canary/0.log"
Jul 28 18:57:10.477894 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:10.477860    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6ffmn_21b940d6-a972-4a7e-b3e2-3054c9bac62f/kube-rbac-proxy/0.log"
Jul 28 18:57:10.498985 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:10.498958    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6ffmn_21b940d6-a972-4a7e-b3e2-3054c9bac62f/exporter/0.log"
Jul 28 18:57:10.520014 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:10.519986    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6ffmn_21b940d6-a972-4a7e-b3e2-3054c9bac62f/extractor/0.log"
Jul 28 18:57:13.277382 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:13.277353    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-r9kc5"
Jul 28 18:57:13.585388 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:13.585334    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-659b67c655-7r4r7_775fa999-365a-46d5-a31d-17ddbd90f20a/manager/0.log"
Jul 28 18:57:14.294539 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:14.294508    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-cd4d44c67-d9v9x_78ed5dcf-3cee-4ad7-965f-811fe00995b0/manager/0.log"
Jul 28 18:57:16.099904 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:16.099872    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qf8qs_621a10bc-44cc-4010-97c5-619c9bbd5c3b/s3-init/0.log"
Jul 28 18:57:16.117752 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:16.117711    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qf8qs_621a10bc-44cc-4010-97c5-619c9bbd5c3b/download-hf-model/0.log"
Jul 28 18:57:16.162521 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:16.162492    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-g7wgt_0e178712-6841-49b8-993e-c02a373c7b17/seaweedfs/0.log"
Jul 28 18:57:21.174469 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:21.174434    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9t7rs_f496a721-c735-4949-b4a7-12bcbdcbecf0/migrator/0.log"
Jul 28 18:57:21.196743 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:21.196719    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9t7rs_f496a721-c735-4949-b4a7-12bcbdcbecf0/graceful-termination/0.log"
Jul 28 18:57:22.697748 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:22.697713    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sl5f_eca38d07-90a1-4d69-aa43-253bd77a26c6/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:57:22.718247 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:22.718219    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sl5f_eca38d07-90a1-4d69-aa43-253bd77a26c6/egress-router-binary-copy/0.log"
Jul 28 18:57:22.738343 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:22.738318    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sl5f_eca38d07-90a1-4d69-aa43-253bd77a26c6/cni-plugins/0.log"
Jul 28 18:57:22.759052 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:22.759025    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sl5f_eca38d07-90a1-4d69-aa43-253bd77a26c6/bond-cni-plugin/0.log"
Jul 28 18:57:22.779969 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:22.779947    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sl5f_eca38d07-90a1-4d69-aa43-253bd77a26c6/routeoverride-cni/0.log"
Jul 28 18:57:22.799697 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:22.799679    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sl5f_eca38d07-90a1-4d69-aa43-253bd77a26c6/whereabouts-cni-bincopy/0.log"
Jul 28 18:57:22.820037 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:22.820014    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sl5f_eca38d07-90a1-4d69-aa43-253bd77a26c6/whereabouts-cni/0.log"
Jul 28 18:57:23.166245 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:23.166214    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rrq8b_9873ffe2-365c-443c-bdb4-582312b3cd12/kube-multus/0.log"
Jul 28 18:57:23.277048 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:23.277022    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zvdqk_20e8d68a-03b6-46f5-8500-1fd50954ac27/network-metrics-daemon/0.log"
Jul 28 18:57:23.295942 ip-10-0-137-238 kubenswrapper[2579]: I0728 18:57:23.295905    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zvdqk_20e8d68a-03b6-46f5-8500-1fd50954ac27/kube-rbac-proxy/0.log"